Getting Blendshapes: How to Bring Face Tracking Data From VTubeStudio Into Your Own Program

9 months ago
5

This is my documentation.

00:00 Introduction
00:21 Part I: Network Communication
01:10 Computers are connected
02:11 How VTube Studio Communicates
03:29 Timing
05:30 Part II: This is Graphic Stuff
07:03 cgltf to the rescue
07:28 "Shape Keys" (in Blender)
09:12 (cgltf) Data Structure (Attributes/Hierarchy)
10:24 Side note about Buffers, Vertex Buffers
11:11 The Hierarchy Comparison
11:32 Models in Raylib
12:35 Part III: Blendshapes (finally)
14:37 glTF Extensions
15:38 How Blendshapes are Computed
16:09 The Loop
17:12 Most applications, ever.
18:49 Optimizing: The Beginning
19:31 Part IV: Multithreading
21:10 Closing Remarks & Bug Fixes
22:02 I'm Done!
22:22 Future Directions

Send Message to Port 21412: {\"messageType\":\"iOSTrackingDataRequest\",\"sentBy\":\"SoftwareName\",\"sendForSeconds\":1,\"ports\":[50506]}

OpenMP (in Visual Studio): https://learn.microsoft.com/en-us/cpp/build/reference/openmp-enable-openmp-2-0-support?view=msvc-170

VSeeFace: https://www.vseeface.icu/
VTubeStudio Wiki: https://github.com/DenchiSoft/VTubeStudio/wiki
Sending Data to VSeeFace: https://github.com/DenchiSoft/VTubeStudio/wiki/Sending-data-to-VSeeFace

glTF Morph Target Spec: https://github.com/KhronosGroup/glTF-Tutorials/blob/master/gltfTutorial/gltfTutorial_017_SimpleMorphTarget.md
glTF No Morph Targets, use Extensions: https://github.com/KhronosGroup/glTF/issues/1036

Raylib: https://github.com/raysan5/raylib/tree/master
RayGUI: https://github.com/raysan5/raygui/blob/master/src/raygui.h#L2935

Assigned TCP/UDP Ports: https://en.wikipedia.org/wiki/List_of_TCP_and_UDP_port_numbers
cGLTF: https://github.com/jkuhlmann/cgltf/blob/master/cgltf.h
JSMN C JSON parser: https://github.com/zserge/jsmn
time.h: https://cplusplus.com/reference/ctime/
Visual Studio Debugger is sometimes wrong, apparently: https://developercommunity.visualstudio.com/t/debugger-shows-data-with-incorrect-struct-members/1225526

Loading comments...