Google Books (or similar) all book scans – $200k bounty (2025)

Join the discussion

• Hacker News

•     If you shouldn't be able to copyright GRAPES...you shouldn't be able to copyright BOOKS.

  by tolerance
• Monsanto here, tell us more about copyrighting grapes!
  by gpm
• That's why they're grapes of wrath. Look for them in the grapes section.
  by TurdF3rguson
• And why's that? I don't see how either of those two things relate to one another. A grape is a naturally occurring fruit, while the creation of a book is wholly a human creative endeavor.
  by ajcp
• Can't LLMs now just write books that aren't publicly available?
  by jmakov
• Comment by Borja is a great example of eternal September.
  by rvba
• Another source I'd love to see scraped or opened up is the New York Times archive, along with other newspaper archives.
  by vagab0nd
• The US should just find a way to quietly share literature access with the Russians, rather than letting piracy be promoted and facilitated for US consumers as freedom-fighter "archiving".

  Between all the piracy, and all the AI training and the purchase/visitor-circumventing AI services, the practice of writing and publishing genuinely good work is being wiped out.

  We're killing the goose that lays the eggs, for selfish gain.

  by neilv
• >We're killing the goose that lays the eggs, for selfish gain

  We already did that when the internet collectively agreed decades ago that everything digital should be free for anyone.

  We're now 20 years downstream of ad-blocking being a virtuous good, and piracy being the ultimate show of liberty, and now suddenly everyone cares about the creator's revenue stream.

  The mask slipped and unsurprisingly the internet is a bunch of selfish morally stunted children. Some of them even pushing 50 years old.

  Yes, I am talking to you with the 4TB of pirated content, proud of not loading any ads in the last 15 years, and getting enraged over LLM training.

  by WarmWash
• AI publishing is just email spam, but for books. When the cost of creating worthless text is low, people do it.
  by inigyou
• Possibly but this act of governmental self-harm is useful to The People. We live in a world where if your valuation is ~1T you can more or less just do what you like. And the work of The People is stolen from you and launderd.

  In such a world, isnt it useful that governments are stupid enough to give adversaries reasons to undermine it? When the government props up a corporate tyranny domestically, and racketeering, should we make a temporary alliance with all its enemies?

  (Eg., the provision to AI companies of all corporate secretes and competitive practices via prompts, eventually to be used against their capital interests and their labour interests).

  by mjburgess
• Russians will just share it back (I’m saying that as a Russian). And if not Russians, then somebody else will.

  What you can do is make sure people can pay you easily, and not put (a lot of) hurdles in your readers way. And when people can’t afford to pay... maybe let them enjoy your work still, and you’ll get a couple more loyal fans who would pay you when they’re able to.

  At least this was my world view before AI has arrived and ruined^W disrupted everything. Now I’m not so sure.

  by notpushkin
• Do you have stats on that?

  I’m not sure piracy or AI training are really affecting book publishing dramatically. But if you have data, I’d be curious to see it. AI scraper bots are a total pain for online publishers and FOSS sites, but AFAIK they’re not really harming book publishing directly.

  The consolidation of publishers and Amazon’s own practices are probably worse for authors than “piracy”.

  by jzb
• >the practice of writing and publishing genuinely good work is being wiped out.

  Most of the best literature in the English language was written before modern IP law was even a thing. There's very little good literature written by authors primarily motivated by money.

  by logicchains
• This ship has sailed for academic publications, and academics define that term very liberally because we want to read everything, fiction included. The shadow libraries started off as a way for scholars in ex-Soviet countries in particular (but also India, SE Asia, etc.) to access literature that simply wasn’t available in their country. But the shadow libraries proved so successful and convenient that academics in all countries are using them now, even if they have access to official subscription services. I use AA several times a day and so do the researchers around me in my office; at conferences, if the presenter mentions an interesting publication, the whole room immediately opens AA on their laptops, etc.

  Even if projects like AA didn’t have nation-level support, academics would find a way to keep as much of it as possible going. After all, we’re the ones who compiled the bulk of pre-2020 material, and we’re the ones who do all the hard work of scanning from our institutional libraries stuff that doesn’t exist anywhere in digital form.

  by TFNA
• Curious as to how you would approach this. I have no experience in this area, anyone on this forum willing to share their expertise?
  by OrangeDelonge
• If it works as AA seems to theorize, you'd need to:

    (a) work out how Google books exposes fragments of books, and see if there's a systematic way of using this to get whole books.  For example, a naive approach might be to find any fragment of the book by searching some exact phrase.  Then, you can search for an exact phrase from the start or end of the fragment it gave you, hoping it will show you the previous or next part of the book.  You can then just loop that to get the whole book.
  
    (b) once you have (a), you need a way of bypassing Google's bot detection/rate limiting.  I don't know what current state of the art is, but there may be a solution for sale out there.  E.g. you pay to receive a cookie or browser state, and use that to fetch the URLs from (a).  Or if you're good/already in the scene, you could do this part yourself.

  by 0x3f
• One of my hopes is that when the AI bubble bursts, some brave person will sneak out a copy of the last frontier model.
  by ThrowawayTestr
• Prediction markets can solve this.
  by thx67
• If it's a bubble, why do you care about frontier models?
  by fastball
• which will be very difficult to run unless you have a large budget to operate your own mini datacenter
  by zuzululu
• Not worried about that, you will only have to wait 3-6 months and get a Chinese model just as good.
  by Aboutplants
• There was a time where you would get a random page preview, some artists found a way to extract full books that way (F.A.T lab?).
  by thenthenthen
• Micropayments (charged in mills, not cents) is the solution. Downloading books remains essentially "free" for the individual but the Internet scale is such that authors would receive compensation for their work. The Spotify model is better than downright piracy. It is very difficult to compete with free.
  by pseingatl
• I would argue precisely that downright piracy is better than the Spotify model. It is based in micro-micropayments, so much so that even at internet scale very few artists outside of the uber-macro-Taylor-Swift size get proper compensation. Sending a single dollar through Bandcamp amounts to hundreds of listens on Spotify.

  It really sucks, but I'd rather pirate and know I'm at fault with the artist – maybe I'll buy some tracks off Bandcamp to make up for it – rather than let Spotify cover the transaction with a legal blanket, while the artists get almost nothing in return.

  by ilikegreen
• Gemini should be trained on those books already, so in theory it could regurgitate some verbatim fragments (as NYT lawsuit against OpenAI showed some time ago).
  by alkyon
• Gemini, gpt and fable are actually very good compressions of internet content. But is lossless compression as in they kept the most important part (for them to fulfill the next token task) and found a way to mimic the rest.
  by motoboi
• Just do it and be legends, Larry. ;)
  by leoc
• Apple won't even help Asahi linux even though it would help hardware sales and give them a ton of goodwill.
  by Cider9986
• So AA is a front for openai?
  by FerritMans