The final version of the 0.01 model has been added. V0.02 is due february with a new architecture.
You MUST update all .py as the old ones wont work with the latest model (many changes involved in the model over this research period!)
This version is the best so far but struggles with dimensions at the furthest points from the mouth, some dimensions are zero'd out to keep things smooth.
Talk to a NeuroSync prototype live on Twitch : Visit Mai
The NeuroSync Local API allows you to host the audio-to-face blendshape transformer model locally. This API processes audio data and outputs facial blendshape coefficients, which can be streamed directly to Unreal Engine using the NeuroSync Player and LiveLink.
- Host the model locally for full control
- Process audio files and generate facial blendshapes
To generate the blendshapes, you can:
- Download the model from Hugging Face
- Apply for Alpha API access if you prefer not to host the model locally.
To stream the generated blendshapes into Unreal Engine, you will need the NeuroSync Player. The Player allows for real-time integration with Unreal Engine via LiveLink.
You can find the NeuroSync Player and instructions on setting it up here:
Visit neurosync.info for more details and to sign up for alpha access if you wish to use the non-local API option.