Case Studies @ HJ
LSDub: Lip-Sync Without Re‑Rendering
An open-source ComfyUI node pack that re-syncs a singer's own mouth to a new song in AI music videos, measured shot by shot against the vocal, b…
Client Humanjava Enterprises (own product), entered in Comfy's Open Source Higgsfield challenge
Role Creator, product owner and lead engineer: pipeline design, the measurement and QA system, the node pack, releases, and the orchestration of AI makers and checkers.
Timeline October 2026 to November 2026
Stack ComfyUI, Python, MediaPipe face mesh, Whisper + forced alignment, LTX-2.5, Qwen-Image-Edit, ACE-Step (open weights), GitHub Actions + Comfy Registry
AI video tools give you performers who look the part but don't sing the lines, and re-rendering a shot throws away everything that was already good. LSDub keeps the take: it warps the singer's own lips, teeth and tongue to the lead vocal, shot by shot, then checks every shot against the vocal before it ships. I designed the pipeline and its measurement system and shipped it as an open-source ComfyUI node pack, building it with AI agents in separate maker, checker, confirmation and QA roles, where no change landed without an independent check.
Don't paint a mouth. Warp the one that's already there.
LSDub is Humanjava's entry in Comfy's Open Source Higgsfield ComfyUI contest: open weights only, no partner APIs, no closed models. Submissions closed on November 3, and Comfy announces the winners on November 10. You can watch it run on the project site and get it on GitHub.
The problem: the shot is right, the mouth is wrong
A multi-shot music video is not one face in one clip, and most lip-sync tools are built for a single talking head.
- Generative lip-sync repaints the lower face, so identity, teeth and lighting drift from the shot you already liked.
- A video holds crowds, wide shots and microphones. Only the lead singer's mouth should move, and only where it can be seen.
- Harmonies and backing vocals sit in the same mix. They must not open the singer's mouth.
- Nothing told you which shots worked. You found out by watching, one shot at a time.
Keep the take, change the mouth, and prove every shot against the vocal.
The approach: warp the singer's own mouth, then measure it
Most lip-sync tools, LTX's LipDub IC-LoRA and the talking-head models among them, generate new frames from the audio or swap the face outright. LSDub does neither. Every pixel on screen is the clip's own, moved to fit the audio frame by frame. It splits a video into shots, finds and names the singer in each, and triages every shot as close-up, medium, or not this time, with the reason. It isolates the lead vocal, aligns every word and sound, and plans a mouth-opening curve from it. Then it warps the mouth already in the take to that plan, using the singer's own mouth shapes, and pastes back only the feathered lower face. Inputs are slots: bring your own clip and song, or generate them with open-weights models.
1. Score against the truth, not the plan
Early QA compared each render with the plan it was painted from, so it passed shots whose lips visibly missed the music. Every shot is now read against the lead vocal itself, on every frame: in sync, no more than a frame late, and the mouth shut in the silences. A shot that fails goes back to the original footage.
2. A checker must not share the maker's blind spot
The verifier judged mouth movement against the original take's own range, so soft and held sung lines looked like failures. Scoring against how much the song actually asks for corrected 18 of 64 verdicts. The three still held back really do under-deliver.
3. Four roles, never one review
Every change went through a maker, a checker who disputes the reasoning, a confirmation that reproduces it from a clean copy, and QA that checks it fits everything else. The judge is frozen before work starts. On the verifier fix alone, the independent checker found three real bugs before release.
4. Every new behaviour has an off switch
Each change to how the mouth moves ships with a switch in the node UI, so the previous behaviour is always one choice away and a creator can compare the two on their own footage.
Results
- 0.1.11 public release (Apache-2.0 node pack on GitHub and the Comfy Registry, with four example workflows)
- ~1,500 engine tests (run in CI on every push, alongside a frozen sync judge)
- 18 of 64 verdicts corrected (fixing a biased checker moved the test set from 33 PASS / 31 REVIEW to 51 / 13)
Released as an open-source ComfyUI node pack (Apache-2.0) on GitHub and the Comfy Registry, at version 0.1.11.
Entered in Comfy's Open Source Higgsfield challenge, with a two-minute demo and a public project site.
A 64-clip test set (4 faces, 4 sets, 4 songs) re-rendered on every release that changes the mouth.
The same maker, checker, confirmation and QA process now runs other builds end to end, from a prompt to a live site.
Inside LSDub
Reflection
I am proudest that LSDub doesn't trust itself. It measures every shot against the vocal, and the process that built it assumed the checker could be wrong too. When the tests said yes and my eye said no, the fix was always to measure against the truth instead of the plan.
