SpecMaskFoley is a sound generation model that can generate audio in real time while keeping it temporally synchronized with input video. It is designed to produce sounds that match the content of the video, including not only continuous audio such as ambient or background sounds, but also short, event-like sounds such as a glass falling and breaking on the floor. Its key characteristic is that it takes video temporal synchronization features as input to the model, which enables alignment between the video and the generated sound. SpecMaskFoley is referenced through an arXiv paper and a demo page, and that it serves as the basis for a real-time sound generation system.
SpecMaskFoley
Generated2026/4/15 — claude-code-refine