FaceSync - Audio Lip Sync & Facial Animation

Asset description

NEW IN 1.4.0 — Audio Localization Manager Keep every language paired with its own baked facial animation and source audio. Localized dialogue assets, explicit default-language fallback and an optional manager work with your existing FaceSync Player. Changing language affects the next requested line; invalid requests preserve current playback. Includes inspector validation, a resolution preview without playback and a complete setup and C# guide. No Unity Localization dependency. This release covers baked dialogue playback; automatic Timeline localization and Addressables loading are separate future work. NEW IN 1.3.0 — Batch Baking Utility Bake selected audio clips or entire folders from one Editor window. Preview the queue, skip clips with existing output, cancel safely, and resume. Create new takes without overwriting existing animation assets. Batch baking guide. Welcome to Frostember FaceSync, the ultimate audio-driven facial animation ecosystem for Unity. Whether you are building AAA cinematic cutscenes, highly optimized VR games, or live VTubing avatars, FaceSync delivers accurate lip-sync and procedural facial movements with zero hassle. Stop wasting hours manually animating blendshapes or fighting with heavy real-time CPU loads. FaceSync bridges the gap between high-end realism and perfect optimization. DOCUMENTATION + ROADMAP | FORUM | YOUTUBE WE HIGHLY RECOMMEND INSTALLING THE TEXTMESHPRO ASSET TO VIEW THE DEMO SCENE TEXTS! THE PLUGIN DOES NOT CURRENTLY INCLUDE FULL MAC OS SUPPORT! 🔥 Key Features Zero-CPU Offline Baking: Real-time audio analysis kills frame rates on Mobile and VR. Our built-in Baker simulates the audio at 60 FPS in the Editor, baking visemes, procedural head noise, and blinks into a highly optimized FaceSyncAnimationAsset. Play it back without runtime audio analysis. Playback and blendshape application still use CPU. Native Timeline Integration: Drag and drop your baked data into our custom FaceSync Track. Smoothly blend animations, scrub through time, and dynamically override eye and head behaviors directly from the Timeline Inspector. Procedural Eyes & Head Motion: Bring dead stares to life. FaceSync automatically calculates realistic blinking, saccadic eye darts, and micro-jitters. The head and neck procedurally react to the character's voice intensity using Perlin noise. 1-Click ARKit & CC3/CC4 Setup: Stop manually assigning dozens of blendshapes. Our VisemeMappingProfile features 1-click auto-generation for standard ARKit (52 blendshapes) and Character Creator rigs, creating highly accurate mouth shapes instantly. Live Microphone (VTuber Ready): Switch to Live Mode and use the included Frostember_MicrophoneInput component to drive your character's face in real-time using any connected microphone. Perfect for multiplayer voice chat and VTubers. Setup Wizard: Get your character fully rigged and talking in under 30 seconds using the automated Setup Wizard. Emotions profiles: A dedicated, robust module inside FaceSyncController that allows developers to seamlessly blend custom facial expressions (e.g., Happy, Sad, Angry) with active lip-sync data. IK Look Tracking: Automatically make your character track any object or player in the scene! The robust inverse kinematics system seamlessly blends with procedural speaking noise and animations, giving your character a dynamic focus without unnatural rotation snapping. The Language Lab Integration The Language Lab by GP Gear Worx Productions includes optional FaceSync integration for its Dialog Sequencer. Localized dialogue lines can drive either real-time or baked FaceSync lip sync, while The Language Lab manages the corresponding subtitles and dialogue audio. After installing both assets, enable the TLL_FACESYNC scripting define through The Language Lab. The Language Lab is a separate third-party asset and is not included with FaceSync. 💻 API & Scripting Triggering dialogue dynamically via code is incredibly easy using the FaceSyncPlayer component. Just assign the baked asset and call .Play(). Perfect for custom RPG dialogue and quest systems. Questions? Audio Localization Manager (1.4.0): FaceSyncLocalizedDialogue stores locale-to-baked-asset mappings. FaceSyncAudioLocalizationManager.TryPlay resolves the selected locale and validates the paired data before changing an existing Player. Locale matching trims spaces, ignores case and treats underscores as hyphens. Fallback uses only an explicit default; duplicate keys and invalid exact matches are rejected. Requires active Player and AudioSource, already-loaded source audio, positive AudioSource pitch and AudioSource Loop disabled. Player Loop remains available. No required Unity Localization, Addressables or paid dialogue-system dependency. Existing animation format and native analyzer are unchanged. Rebake recordings after editing their audio. Automatic Timeline localization, coordinated pause/resume and asynchronous loading are outside this release. Batch Baking Utility (1.3.0): Tools > Frostember Studios > FaceSync > Batch Baker. Editor-only, sequential processing of imported clips or folders, recursive scanning, duplicate removal, read-only preview, per-clip results and cancellation. Existing output is skipped by source AudioClip reference by default; optional new takes always use unique asset paths. Batch input: mono/stereo AudioClips, at least 1/60 second, Load Type = Decompress On Load. Import settings are not changed. Uses the existing 60 Hz baker and data format; no runtime migration. Core Architecture & Features: Audio Processing: Utilizes the Oculus LipSync engine for highly accurate real-time FFT phoneme detection (15 core visemes). Offline Baking System: Custom Editor Window (FaceSyncBaker) bakes complex audio analysis and procedural animation at 60 FPS into a lightweight ScriptableObject (FaceSyncAnimationAsset). Zero-CPU Playback: The FaceSyncPlayer component reads baked assets, eliminating runtime CPU cost for audio analysis, making it highly optimized for Mobile and VR. Native Timeline Integration: Custom Playable API implementation (FaceSyncTrack, FaceSyncClip, FaceSyncTrackMixer) supports smooth blending, editor scrubbing, and dynamic state overrides. Procedural Animation: Mathematical real-time generation of saccadic eye darts, micro-jitters, and voice-energy-driven Perlin noise for head and neck rotations. Auto-Mapping Profiles: VisemeMappingProfile includes algorithms to auto-generate mapping for ARKit (52 blendshapes) and Character Creator 3/4 standards. Live Input: Included Frostember_MicrophoneInput script captures system audio for real-time processing (VTubing/Multiplayer). Performance & LOD: Built-in distance-based Level of Detail (LOD) and frustum culling automatically disable processing when the mesh is off-screen or far away. GC Optimized: Runtime playback loops and Timeline mixers are built using array-based iterations to prevent Garbage Collection allocation spikes. C# API: Clean and accessible API for triggering dialogue dynamically via code (.Play(), .Stop()). Dependencies & Requirements: Requires the free Oculus LipSync Unity Integration package (Meta SDK). => Included in package Compatible with Built-in, URP, and HDRP render pipelines. Works with any Humanoid or Generic rig that utilizes SkinnedMeshRenderer blendshapes.