What happened to TheNumbers.com

Join the discussion

• Hacker News
• I have small website with archive of older radio broadcasts mostly in Russian. Lately, 90% of traffic comes from USA ;)

  Previously it was like 5%.

  URL is http://radar.lv btw

  Thanks God i can serve up to a terabyte per month of traffic easily, otherwise it would be a catastrophe. I am not against bot scraping, but I worry about stability for meat visitors.

  So, I think of enabling payments for website visits cloudflare recently developed.

  I also have the problem of old technology like the mentioned site. While my personal blog uses static generator, archive website uses ancient Drupal version, which has no security patches for many years already.

  by theragra
• Just FYI, the bigger companies all allow you to block crawlers via robots.txt:

      # Block Anthropic (Claude)
      User-agent: ClaudeBot
      Disallow: /
      User-agent: Claude-SearchBot
      Disallow: /
      User-agent: Claude-User
      Disallow: /
  
      # Block OpenAI (ChatGPT)
      User-agent: GPTBot
      Disallow: /
      User-agent: OAI-SearchBot
      Disallow: /
  
      # Block Perplexity
      User-agent: PerplexityBot
      Disallow: /
  
      # Block Google's AI Training
      User-agent: Google-Extended
      Disallow: /
      User-agent: Google-Extended-Factual
      Disallow: /
  
      # Block Microsoft's Search & AI Crawler
      User-agent: Bingbot
      Disallow: /

  by djoldman
• What a throwback. Back in 2015 I have started Applaudience, which at the time was the only provider of real-time cinema ticket sales data. I still remember comparing our numbers against TheNumbers.com as part of calibration. I have since moved on to other businesses, but this remains one of my favorite pieces of technology that I have developed. Would love to bring it back one day.
  by gajus
• One thing I’m wondering is whether we need an open source set of technical patterns and libraries for dealing with this changing traffic mix.

  Millions of small sites and creators don’t have the ability to design their own protections against large scale automated access. If useful content now attracts aggressive crawler traffic, many sites will be too expensive or unreliable to run.

  Is part of the answer a community response? Perhaps a community-maintained toolkit, based on traffic data, that host sites can apply? It could include standard agent identification, rate limiting, traffic classification, access policies, caching, challenge mechanisms, logging, attribution and usage control etc etc.

  In effect, we need much stronger road rules for today’s automated traffic, available as open technical patterns and libraries rather than every site owner having to invent this alone (they won’t).

  by baud9600
• At the risk of oversimplifying things from a distance, this site -- especially the free, public-facing part of it -- seems like it would be an ideal candidate for a rewrite using static site generator/framework. That, coupled with a bot-aware CDN should keep them online at a reasonable cost for many years to come.

  Otherwise, I'm very curious to know more about their old and new architecture and what sorts of mitigation/scaling strategies they've started using to keep the site online.

  by ethagnawl
• A couple years ago, I was running a website which allowed the public to view all the US government handouts to small businesses during the COVID-19 pandemic. It also tracked all the fraudulent loans being prosecuted by the DOJ, and allowed anyone to run structured queries over the public dataset. There was a "donate" button which took in ~$2k in donations over the lifetime of the site, and you could download the entire underlying dataset (around 10 GB uncompressed) for free directly on the site.

  Despite the "download all data" link being prominently placed on the front page, the AI scrapers decided it would be more efficient to download terabytes upon terabytes of raw HTML by paginating through every possible facet on the search endpoint. Even with CloudFront caching results and a fairly efficient backend setup, the monthly bill ended up with around $1k just going toward network ingress/egress, so I shut down the site the following month.

  by primitivesuave
• I think people are missing one of the points of the article. It's not just that agents are hammering the site, it's that there might be lurking vulnerabilities that allow malicious usage, which is why it went down, then came back up with a fraction of the data and a reduced design.

  The article says (speculates?) that malicious users are trying to get privileged access for an edge in prediction market betting. From the article:

  > If you could see The Numbers data before everyone else, every single week, you would have a significant edge over all the other traders - learning the answers slightly ahead of publication would allow you to front-run the trades.

  by abetusk
• "One Reddit theory even suggested it was a deliberate rug pull designed to cripple the free site to push people towards paid products."

  I know this isn't really the point of the article, but I've been thinking about this a lot. I wonder if we're going to see more resources go this way.

  I used to publish little doo-dads as opensource software. Not because it was something that was legitimately ground breaking or anything (it absolutely wasn't) but because I had a problem, and I thought "heh wouldn't it be cool if someone else had a similar problem and could use my resource for it."

  But now I'm really reluctant to give more stuff to the free web. Because the fact that it gets scraped and added to a pile of training data to later be monetized really rubs me the wrong way.

  And I can see why maintainers of sites like this, or other free but incredibly useful resources might start to get irate at that.

  by podgietaru