Discussion summary
A developer created a tool enabling LLMs like Claude to 'see' videos, addressing limitations of existing models. The project has received positive feedback and comparisons to other solutions like Gemini.
What the discussion says
- Some users find the tool innovative and useful.
- Concerns about cost and practicality are raised.
- Comparisons are made with other models like Gemini.
“Claude won't accept video files, ChatGPT reads transcripts only.”
“I gave Claude a video for a speeding ticket, and it was spot on.”
Comments
Hacker News
Are models any good at descerning motion from multiple frames?
For instance if I gave models multiple animations of a bouncing ball as individual frames. Would they be able to tell which bounce was the more realistic motion.
(Is this a potential new benchmark? maybe also variations of stair dismount)
by Lerc
The comments in this post strongly validate the need for reliable video processing and understanding with VLMs.
While you can use Gemini or other local VLMs, the real challenge is token efficiency, accuracy, and coverage. For example, how do you make a VLM “watch” a 2-hour or 4GB video without losing context or meaning?
Video transcript alone can be sufficient for basic workflows needing no visual context. But when deep contextual understanding is required, e.g., self-driving, security analysis, warehouse tracking, etc., you’ll need more advanced methods like keyframe sampling, clipping, chunking, and shots+transcript.
You can explore the different encoding strategies we designed for efficient video processing and understanding here: https://vlm-run.github.io/mm/encoders/#video.
FYI, the repo is now public, and contributions are welcome.
by cpnwaugha
Nice @OP i put together something similar as well. Incidentally I found for motion design specifically llm is not able to infer specific animations as well as it just being described very plainly and accurately what is happening and the timing.
One thing which sort of worked decently was actually take the frames and put them into a grid and have the agent look at the image of all of the frames together. It did surprisingly well but missed a lot of subtle details that it couldn’t see.
Also tried various kinds of vision embeddings, heat map of motion etc, and blur etc to show motion. But none really worked as well so I ended up just describing it until it got it. Haven’t quite found the right solution yet.
by gvkhna
I was creating a scene by scene remake of a cutscene from an old DOS game. The sprite sheet had several sprites which were cycled (e.g. a horse with it's head down and up). The engine would cycle through these regularly to create some "liveliness" in the background. It was tedious and I didn't want to figure out which sprites belonged at which pixel location.
I recorded a video of the relevant part of the cutscene using dosbox and then split it into numbered frames using ffmpeg. Then I gave that + the spritesheet to Claude Code and asked it to figure it out and tell me which ones are at what position. I should probably have deduped it but in any case, it churned through the whole thing and got one or two out of 15 or 16 sprites right. The rest, it just dropped into random places. YMMV
This looks cool but this should be renamed without having Claude in the name.
I’m currently punishing Fable by making it watch the entire series of 7th Heaven.
by nickpeterson
"Where the video goes: stays on your machine" - No, the frames (that this tool extracts) obviously get sent to Anthropic if you use Claude.
by bonoboTP
Pretty terribly expensive way to watch a video with Claude.
Use Gemini or some local VLM to do this way more efficiently. We spent quite a bit of time on video understanding, and Claude will just burn tokens.
Check out this library: https://vlm-run.github.io/mm/
You can swap models and try out different encoding methods for videos (https://vlm-run.github.io/mm/encoders/#video)
Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- Are models any good at descerning motion from multiple frames?
For instance if I gave models multiple animations of a bouncing ball as individual frames. Would they be able to tell which bounce was the more realistic motion.
(Is this a potential new benchmark? maybe also variations of stair dismount)
by Lerc - The comments in this post strongly validate the need for reliable video processing and understanding with VLMs.
While you can use Gemini or other local VLMs, the real challenge is token efficiency, accuracy, and coverage. For example, how do you make a VLM “watch” a 2-hour or 4GB video without losing context or meaning?
Video transcript alone can be sufficient for basic workflows needing no visual context. But when deep contextual understanding is required, e.g., self-driving, security analysis, warehouse tracking, etc., you’ll need more advanced methods like keyframe sampling, clipping, chunking, and shots+transcript.
You can explore the different encoding strategies we designed for efficient video processing and understanding here: https://vlm-run.github.io/mm/encoders/#video.
FYI, the repo is now public, and contributions are welcome.
by cpnwaugha - Nice @OP i put together something similar as well. Incidentally I found for motion design specifically llm is not able to infer specific animations as well as it just being described very plainly and accurately what is happening and the timing.
One thing which sort of worked decently was actually take the frames and put them into a grid and have the agent look at the image of all of the frames together. It did surprisingly well but missed a lot of subtle details that it couldn’t see.
Also tried various kinds of vision embeddings, heat map of motion etc, and blur etc to show motion. But none really worked as well so I ended up just describing it until it got it. Haven’t quite found the right solution yet.
by gvkhna - I was creating a scene by scene remake of a cutscene from an old DOS game. The sprite sheet had several sprites which were cycled (e.g. a horse with it's head down and up). The engine would cycle through these regularly to create some "liveliness" in the background. It was tedious and I didn't want to figure out which sprites belonged at which pixel location.
I recorded a video of the relevant part of the cutscene using dosbox and then split it into numbered frames using ffmpeg. Then I gave that + the spritesheet to Claude Code and asked it to figure it out and tell me which ones are at what position. I should probably have deduped it but in any case, it churned through the whole thing and got one or two out of 15 or 16 sprites right. The rest, it just dropped into random places. YMMV
- This looks cool but this should be renamed without having Claude in the name.
- I’m currently punishing Fable by making it watch the entire series of 7th Heaven.by nickpeterson
- "Where the video goes: stays on your machine" - No, the frames (that this tool extracts) obviously get sent to Anthropic if you use Claude.by bonoboTP
- Pretty terribly expensive way to watch a video with Claude.
Use Gemini or some local VLM to do this way more efficiently. We spent quite a bit of time on video understanding, and Claude will just burn tokens.
Check out this library: https://vlm-run.github.io/mm/
You can swap models and try out different encoding methods for videos (https://vlm-run.github.io/mm/encoders/#video)