Birbla

Join the discussion

Write your take first — we'll ask for email only when you're ready to publish.

Sign in
  • Hacker News
  • Sometimes I think people forget how capable computers are. 500k is not much. You can just slap that in a Lucene instance. This is a solved problem.
    by pimlottc
  • Sorry I have a lot of trouble understanding what this is useful for. Like, I am never going to replace it with Google, DuckDuckGo, ChatGPT or even Bing.
    by fg137
  • This is actually where I see software going in the short term -- cloud moving to local.

    A few years ago, if you wanted translation, you'd use Google Translate. If you wanted to search the web, you'd use Google search.

    But for a few gigabytes, you can now install nllb-200-distilled-600M, and get translations for almost any language locally. You can have your computer crawl the web, create abstracts and categorizations for websites, and build search exactly as you want it.

    The main limiter now is hard drive space (and to an extent, local compute) -- but right now it feels like the 70s again where the terminal into a remote server turned into building applications locally.

    by eggbrain
  • FYI for those needing a list of domains

    Subject: I want all domains and subdomains https://groups.google.com/g/common-crawl/c/XC2QmOE-sdI?pli=1

    or google for COMMON CRAWL

    by frogger8
  • Reminds me that AltaVista's servers ran in 4G of RAM (there were famous, at the time, pics of the circuit boards - DEC was rightfully proud of this, 30 years ago) and that a modern AltaVista should run on a decent laptop :-)
    by eichin
  • TS;DR: Too Sloppy; Didn't Read.
    by headz
  • Interesting project. Website discovery is indeed in a pretty dire spot, definitely a space that needs innovation. An auto-labeled website directory isn't that silly of an idea.

    I have a 400 GB sqlite database with samples of rendered root document DOMs I use for ad detection in Marginalia Search I've been meaning to explore similar ideas using.

    by marginalia_nu
  • Here's my impressions of your algorithm:

    1. read each site

    2. rent a 4090 with https://vast.ai to run vllm

    3. let llm model invent its own category and tag names freely

    4. save 1KB of metadata each

      a. a small local language model that reads each one and writes a name, two or three sentences, a category, and a handful of tags.
    5. `code is going up as open source` soon (TM)
    by iFire

Explore Birbla archives