Birbla

Join the discussion

Write your take first — we'll ask for email only when you're ready to publish.

Sign in
  • Hacker News
  • This is cool. I could see myself downloading the articles behind the first couple pages of hacker news with this, for viewing on a flight or long distance train ride with spotty internet
    by lolpython
  • I've accumulated a bunch of old website archives over the years. The funny thing is the ugly HTML dumps have been more useful than the "perfect" archive.

    It's one of the reasons I've become a bigger fan of RSS over time. A feed from 10-ish years ago is often more usable today than a carefully preserved (application) website.

    by coffeecoders
  • Maybe it is just me, but by far most of the time, when I want to archive something from the internet, it is information and information is best served in an absolutely minimal text format like html or md.
    by couscouspie
  • I have a project for creating and archiving RSS feeds, keeping the full history from the time the crawler starts. I need to clean up a bit, then will open source it soon.
    by tamnd
  • So this is like using wget --mirror except that it works on pages that require javascript, right?
    by rahimnathwani
  • Yeah, it is. For example, openai.com is rendered with Next.js, so I will try to mirror it tomorrow.
    by tamnd
  • Neat project, I like the idea. One thing from a quick read: you launch Chrome with --no-sandbox. Is there a good reason for that? Security wise it's probably not a good idea. If there is no reason, I'd suggest leaving the sandbox on!

    In any case, cool stuff :)

    by dimiprasakis
  • --no-sandbox is needed in docker, maybe they assume it will mostly run in docker?
    by nikisweeting
  • This seems like it has potential to create a lot of load on a site- are there settings to set how fast it clones or avoid images/videos? Is there a way to only get a subset of a website?
    by gregwebs
  • Just pretend you're an AI crawler problem solved
    by ares623
  • Could you help create a new issue for that? I will do it later. It is already 1:00 AM my time, but I am happy that anyone is interested in it. : )
    by tamnd
  • For those with an eReader, one thing that works really well is using pandoc to download and convert a webpage to EPUB that you can then load to your reader.

      pandoc --from html --to epub --output /PATH/TO/FILE.epub https://example.com
    by latexr
  • Thanks, will try this out on the Kobo later.
    by arikrahman
  • I've been using httrack (https://www.httrack.com) to download wikis to read on flights, which isn't perfect but better than I'd found previously. I'll try this out, I'd be delighted to have good results. Thanks for the post.
    by telesilla
  • https://github.com/archiveteam/grab-site or browsertrix may be easier to use for some, it's what was used to save a lot of the data.gov stuff before it got taken down.
    by nikisweeting
  • This brings back memories. Around twenty years ago, internet was still expensive dial-up, so I used to go to an internet cafe, run HTTrack to download websites and manga, copy everything onto my tiny 128MB USB stick (felt very large at that time), then bring it home and read offline ;))
    by tamnd
  • Specifically for wikis, is there a reason you wouldn't use Kiwix? For non "official" releases it's more complicated, but there are some services to generate the ZIM files. The desktop reader app is pretty good in my experience.

    https://wiki.openzim.org/wiki/Build_your_ZIM_file

    EDIT: https://get.kiwix.org/en/solutions/applications/kiwix-reader...

    by throwaway219450
  • > No tracking, no network calls, no surprises.

    Won't comment on a project (though idea seems interesting) but this in README is a tell for me ;)

    by xlii
  • Somehow 'Kage' is the first name claude suggests to me for any new project as well
    by praveer13
  • It's not just no tracking — It's no surprises.
    by xd1936
  • I find SingleFile [0] to be a much more robust version of this.

    It strips out all the JavaScript too, but also packs everything into a single HTML file that is easy to transfer. Binary assets (like web fonts and images) are packed as base64 strings.

    They also offer a CLI powered by Puppeteer. [1]

    [0]: https://github.com/gildas-lormeau/singlefile

    [1]: https://github.com/gildas-lormeau/single-file-cli

    by maxloh
  • This is what I first thought and it's a very elegant solution, and not needlessly overcomplicated.
    by arikrahman
  • What's the difference with, any webbrowser on a computer, File -> Save as ?
    by HelloUsername
  • And thanks for the link. Let me implement this single HTML feature, it looks nice to have!
    by tamnd
  • Love love love SingleFile too. The FF extension works pretty well for a clean save.

    That said, Kage looks promising if OP can combine SingleFile reproduction quality with the HTTPTrack spidering approach. SPA's are kinda tricky with archiving and do wonder how well Kage would handle that

    by wamatt
  • It seems this repo only saves one web page?

    What I'm implementing here is mirroring a whole website, with all its subpages, so you can browse it all offline. For example, all essays from paulgraham.com.

    by tamnd
  • > kage serve $HOME/data/kage/paulgraham.com

    If the result is static why does it need a server? Isn't it possible to make it so that it can simply be opened by the browser? Like:

    $ firefox $HOME/data/kage/paulgraham.com

    Then the result would be useable on machines without kage nstalled.

    by ninalanyon
  • You’ll likely run into a ton of CORS issues doing that.
    by afavour
  • Usually JavaScript is blocked when you load pages that way.
    by doctoboggan
  • You could use python -m http.server instead. I haven't tried it yet, but it should work.

    Actually, Kage has two parts: a crawler that crawls pages and converts them to clean HTML by capturing the DOM after rendering in Chrome/Chromium, and a pack/serve component that packages the result as either a ZIM file for Kiwix or an executable file.

    by tamnd