Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- What models did you use for the stages? I see Qwen2.5-VL-7B-Instruct mentioned as an advanced option, so I assume maybe Qwen2.5-VL-3B-Instruct by default (which is what I also use for a lot of stuff, it is incredibly good at "clean" OCR, but as you maybe indicate not the best at "describing a scene").
EDITED: I didn't realize Whisper was a local model. I never tried transcription before, so I had always figured it was a pay model by OpenAI. I'll have to check it out (although the runtime listed here is a bit daunting).
For that project I'll say I don't see much degradation in embedding quality at much much worse quality than 720p (all the way down to 240p), which speeds things up considerably. Although I don't really do face or object detection, just scene embeddings. To me any process whereby it would take longer to process the video than watch it is probably a no go in general. Obviously a challenge for local-first analysis.
- I will be doing these things with local LLMs
Take a fast, small and powerful LLM running locally to index my personal data like images, videos, documents and enrich them and tag with the enriched metadata.
Want to group by people - Search tagged metadata and group it What to search an image by description - tagged metadata What to organize by anything - tagged metadata
This should (hopefully) put an end to my file clutter
by insumanth - I am in no way a tech savy person, don't know coding, don't know networking or AI much either. But I definitely want to have a system like this. An AI powered gallery / video repository that can help me find moments, people, colors, objects from 100s of 1000s of files.
Local LLMs sound so cool but I know they won't be easy to setup or use for common joe like me.
- I have an RTX 5090 card but it only has 32 GB RAM, can something like this work on my machine?by cake-rusk
- Yes, and it’ll result in much faster results than the ones that I did with my computerby iliashad
- it is possible to use apple gpu with containers. either with podman + runkit + recent mesa or with recent vllm-metal from docker https://www.docker.com/blog/docker-model-runner-vllm-metal-m...by fl0id
- I was looking for a solution for this issue of running docker containers over MPS and utilizing their GPU power. I think this project will be the solution for it, I’ll try it very soon and add support for it. Thank you, much appreciatedby iliashad
- by lgats
- Thank youby iliashad
- I was surprised to learn that the
Do I have it right? Would Windows ARM performance be similar for those cpu?M1 Max CPU is an ARM/SoC, comparable to an 11th gen Intel i9ref: https://www.cpubenchmark.net/compare/4585vs4245/Apple-M1-Max...
by WarOnPrivacy - To your question, I can’t deny or confirm that because I didn’t tried it this project over a Windows machine yet or a machine with this configby iliashad
- No comparison. M1 Max has 400GB/s RAM bandwidth while Snapdragon X2 Elite, the latest and greatest , has 228GB/s RAM bandwidth.by voidmain0001
- “Comparable” is maybe true if we are talking about single core performance, but for memory bandwidth, the M1 Max is about 8 times faster. Wider bus, lower latency, not even close.by owldown
- It's also a bit apples (heh) to oranges for a handful of reasons, but most impactful
- "unified" ram makes all the system ram available as VRAM - dedicated ai coaccelerator thingy
Both of these reasons allow the apple silicon chips to crush conventional cpus in these kind of AI model workload stuffs
No idea about what the windows arm stuff is capable of. I know they use Qualcomm snapdragon chips though.
- Does it work for porn collections too?by Beijinger
- Not sure if you’re being sarcastic but I think this is an interesting question. Would deep seek be useful here since it is local?by supertroop
- Why it’s always the same question? Hahah. I posted my project over Reddit and I got the same one hahahby iliashad
- Asking the important questionsby sarjann
- Last time I tried whisper, it hallucinated an elaborate conversation from sounds of slapping and moaning and it took minutes to spit every single line of it.
- You'll need a lora for this, porn content rejection is heavy. Or you'll need a abliterated model, not sure if vision also works.
You might want to add something like yolo finetune to detect scenes + face recognition too.
by pduggishetti - DaVinci 21 has indexing built-in (AI IntelliSearch). Not to diminish the work you did, but this is now available to many users (probably only Studio users since it has AI in the name)by robrain
- Yes, I didn’t look at it. But does it upload your videos to the cloud or process them locally? And does it allow to provide custom faces data to help labeling faces in your videos ?
I think Adobe premiere pro have it as well but cloud processed
by iliashad - > Then, run the frame analysis pipeline, which will divide the video into separate video scenes (1s each, or 1fps) > (…) > Frames analyzed 57,537
Aha, it makes total sense. This number sounds much more reasonable than “669 GB”, since the actual total size of processed frames would be like 10-30 GB.
(Not downplaying anything. Doing-at-home always requires some math on practicality)
> Total compute time 67h 40m 42s
I’m just curious tho — is there any paying options that can accelerate this kind of process? Just spin up GPU instances?
by esjeon - Yep. Go to vast.ai, spin up a cheap GPU instance, add a bit of code to the project and let it run it finish in just a few hours for like ten bucks.
But it's not as fun as running local model right here on your computer on your own desk. It feels like magic.
by egorfine - > Aha, it makes total sense. This number sounds much more reasonable than “669 GB”, since the actual total size of processed frames would be like 10-30 GB.
The reason why is “669 GB” is the total raw footage size when I'm doing the video processing, I downscaled each frame to 720p to make the video processing much faster and I don't need full original quality in order to get accurate results (as far as I know and experiment with).
> I’m just curious tho — is there any paying options that can accelerate this kind of process? Just spin up GPU instances?
For now, I found that NVIDIA GPU for example RTX 3060 with 12GB Vram was much faster than my M1 Max. (still working on optimizing for speed and accuracy).
by iliashad - Something I've enjoyed more than I expected is Google and Apple photos sending me photo memories and compilations of various things in my life and my kids lives over the last decade.
I'm really bullish on taking more video of my kids, with the thought that it will become easier and easier for AI to put them into little compilations I can enjoy later.
by justinram11 - do you use android and ios, or is there another benefit to having personal media with both?by JMiao
- You don't mind Google using your kids to train their models and advertising algorithms?
Years from now they'll be getting "hey look at BIKE BRANDS' NEWEST CHEAP BIKE REMEMBER WHEN YOU USED TO RIDE BIKE BRAND BIKES"
by goodmythical - I think the Apple stuff is done 100% on device.
Google loves scanning stuff on in the cloud though.
by theshrike79 - I wish I could connect Apple photos to my Spotify account and have photo memories connected with songs I listened to at the time :)by mwelpa