Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- It takes a bit of setup and a huge download, but every time I need a good domain I follow this old post from Derek Sivers. I have Claude de-dupe it and turn it into a searchable database (on my machine), then have it search genres and terms I'm looking for. It's a task Claude is very well-suited to, from the technical implementation to back-and-forth about selections. [link]: https://sive.rs/comby tpowell
- Holy over-engineering, Batman!by thesuitonym
- Note -- if you do this, watch out for requesting access to "all tlds". They send you two emails per TLD -- one for your pending state, and one for your approved/rejected state. I suddenly had 1k+ emails flooding into my inbox, until I found the setting on their website to disable emails.by eggbrain
- Check out my latest project! You can fork it, tweak the policy manually or with AI, run the system and watch the data come in! It's engineered to keep a low data footprint, so 500k domains fits into 1GB on disk. If you have local models it's free! You just might not get the best throughput depending on your GPU. My production data is not exposed anywhere yet, and I may never expose it. The point is for you to fork and make your own policy, and thus your own personal search engine! The article covers basic analysis on my data, so it's worth a read if you're interested! A deeper analysis may arrive with V2 if I ever do itby dreamforever
- Like a personal Google? How do you bypass all the captcha, ip bans, cloudflare turnstile antibot stuff etc?by orliesaurus
- despite this limitation, there is still some good stuff out there, and with the priority steering, you can focus compute on what you actually want, fast and cheap.by dreamforever
- Thats the fun part, the user just went with happy path. Javascript, captchas, cloudflare protected content did not made to the catalogue. This sort of use case exists in LLM training data a lot which makes it easier. The data gathered by the user is not really practically useful cause there are way too many gotchas when it comes to web scraping and building a catalogue (source: I have done scraping for a particular domain data and had to do at least 10+ iterations to get it >90 right)by sandeepkd
- They don't: "skips the model entirely if the page is empty, parked, or a bot-challenge wall"by voidUpdate
- This is a damn good project. Makes me want to make headway into an idea I've had for quite some time P2P search...we'll see.by NetOpWibby
- Sometimes I think people forget how capable computers are. 500k is not much. You can just slap that in a Lucene instance. This is a solved problem.by pimlottc
- Approaching search by just tossing the data in Lucene is how you end up with Confluence's search box though.
- Sorry I have a lot of trouble understanding what this is useful for. Like, I am never going to replace it with Google, DuckDuckGo, ChatGPT or even Bing.by fg137
- I was wondering the same thing. I’ve wished for just a big blob of the web to grep and regex through, but I don’t think this is that much easier than using duckduckgo or even google.by prepend
- It's not for that, sorry, I should have been more specific. It's for people who wanna put in the effort and steer their own crawl to surface their own slice of the web. The article is just a little story of the journeyby dreamforever
- This is actually where I see software going in the short term -- cloud moving to local.
A few years ago, if you wanted translation, you'd use Google Translate. If you wanted to search the web, you'd use Google search.
But for a few gigabytes, you can now install nllb-200-distilled-600M, and get translations for almost any language locally. You can have your computer crawl the web, create abstracts and categorizations for websites, and build search exactly as you want it.
The main limiter now is hard drive space (and to an extent, local compute) -- but right now it feels like the 70s again where the terminal into a remote server turned into building applications locally.
by eggbrain - The number of times we've gone from cloud/server access via terminal to local compute back and forth is something that always makes me laugh a bit.by dylan604
- FYI for those needing a list of domains
Subject: I want all domains and subdomains https://groups.google.com/g/common-crawl/c/XC2QmOE-sdI?pli=1
or google for COMMON CRAWL
by frogger8 - Reminds me that AltaVista's servers ran in 4G of RAM (there were famous, at the time, pics of the circuit boards - DEC was rightfully proud of this, 30 years ago) and that a modern AltaVista should run on a decent laptop :-)by eichin
- During AltaVista's prime, 32MB would have been a lot of memory for a typical computer.by coredog64
- TS;DR: Too Sloppy; Didn't Read.by headz
- Interesting project. Website discovery is indeed in a pretty dire spot, definitely a space that needs innovation. An auto-labeled website directory isn't that silly of an idea.
I have a 400 GB sqlite database with samples of rendered root document DOMs I use for ad detection in Marginalia Search I've been meaning to explore similar ideas using.
- It feels like we've hit a point where search engines can become what "todo list apps" were for devs 10 years ago.
What a homebrewed solution lacks in coverage it excels in indexing and serving a small slice of the internet really really well.
by jeromechoo - I have been wanting do do this. The biggest source of domains is certificate transparency logs. Also ICANN zone files. According to some scientific papers these cover 88% of all registered domains. You could crawl dns for CNAME records with all ipv4 IPs by distributing requests across dozens of DNS servers, the internet archive or the common crawl but doing it for the internet archive is a dick move without giving them money
There's about 200 million active domains currently. That's about 66% of all businesses worldwide of which there are around 300 million. Around 100 to 150 million have active webpages
by alightsoul - Here's my impressions of your algorithm:
1. read each site
2. rent a 4090 with https://vast.ai to run vllm
3. let llm model invent its own category and tag names freely
4. save 1KB of metadata each
5. `code is going up as open source` soon (TM)a. a small local language model that reads each one and writes a name, two or three sentences, a category, and a handful of tags.by iFire - They really needn't have specified "in a weekend" cause yeah we can tell.
Since when has low effort become a selling point anyhow?
by moffkalast - don't forget:
6. let llm write a blog post about this conversation
by johntash - The technical details are on another page: https://alexmorleyfinch.github.io/marlin/history/v1/article/...
Your impressions seem about right, but there are a few control steps it seems.
by jeroenhd - Code appears to already be up: https://github.com/alexmorleyfinch/marlinby iambenm