Video previews in Image History stop working after ffmpeg 9.0 upgrade (-vsync removed)

[julianschill]

Expected Behavior

Videos should show an animated webp thumbnail (or at minimum a static jpg frame) in the Image History pane, like they did before the ffmpeg upgrade. Only an ffmpeg upgrade changed on the system; no SwarmUI version change was made.

Actual Behavior

After upgrading ffmpeg to 9.0, video thumbnails in the Image History pane render as broken/blank images, on both desktop and mobile. Static image thumbnails are unaffected. The browser receives a response with Content-Type: image/jpg whose body is actually the raw video bytes (an MP4 container), which an cannot render.

Steps to Reproduce

1. Run SwarmUI on a system with ffmpeg 9.0+ (e.g. Arch Linux, pacman upgrade from ffmpeg 8.1.2-11 → 2:9.0-5).
2. Ensure Server → UI → "Allow Animated Previews" is enabled.
3. Generate a video (e.g. an .mp4 output via the minimax-h3 backend, or any text2video model).
4. Open the Image History pane.
5. Observe the new video's thumbnail is blank/broken. A static jpg preview file (.swarmpreview.jpg) exists next to the video, but no .swarmpreview.webp.

Debug Logs

No errors in the log

Other

Root cause analysis:

• ffmpeg deprecated -vsync in 5.1 and removed it in 9.0. The webp preview command in src/Accounts/UserImageHistoryHelper.cs (DoFfmpegPreviewGeneration) still passes -vsync 0, so with ffmpeg 9.0 the command exits non-zero and no .swarmpreview.webp is created. The static .swarmpreview.jpg step doesn't use -vsync, so it still succeeds.
• This exposes a second, compounding bug in GetOrCreatePreviewFor (src/Utils/OutputMetadataTracker.cs): when a video has a jpg preview but no webp, the fallback branch runs new Image(data, MediaType.GetByExtension(ext)) with the video's extension (mp4). ToMetadataJpg() returns null for video media types, so the cached preview entry stores null data, and ViewOutput (src/Core/WebServer.cs) then serves the raw MP4 file bytes labeled as image/jpg — which the browser can't display.

Suggested fix:

1. In DoFfmpegPreviewGeneration, replace -vsync 0 with -fps_mode passthrough. Since ffmpeg < 5.1 has -vsync but no -fps_mode, and ≥ 9.0 has -fps_mode but no -vsync, make it version-agnostic (detect version, or try -fps_mode passthrough first and fall back to -vsync 0).
2. In GetOrCreatePreviewFor, fix the fallback so a video with only a jpg preview serves that jpg directly instead of routing it through ToMetadataJpg() with the source file's video media type.
3. After the fix, users with already-cached broken preview entries need to run Reset All Metadata (Utilities tab) or wait for the 24h preview cache to expire so previews regenerate.

Added patches for the suggested changes:
fix_ffmpeg_vsync.patch
fix_preview_fallback.patch

[mcmonkey4eva]

Valid issue report, patched.

LLM generated analysis not valid and not helpful. Please do not copy/paste something an LLM spit out as if it's a genuine message, it just wastes other people's time.

[julianschill]

Sorry for AI slop and thanks for patching the ffmpeg issue. But looking at the commit it doesn't seem you patched the fallback issue, when there is no preview. I haven't looked too deep in the code but the second patch at least showed me still images when there wasn't an animated preview. Maybe you can have a second look, if the LLM analysis was really wrong.

[mcmonkey4eva]

Yes the LLM analysis was really wrong.

I missed that there was a second issue slipped into the long message, fixed now.

[julianschill]

Ok. Will check more carefully next time, then. Thanks for your time and this awesome piece of software.