Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- Note: Article published by Autom.dev which, from a quick read of their homepage, seems like it scrapes Google search results in violation of Google's terms of service and sells those results to customers via an API. That's just my quick read of it, though, so this could be wrong.by JohnTHaller
- Note: Irrelevant.
The reported behavior exists, does it not?
Anecdata: I've observed this behavior for several weeks already as a regular user without a Google account and there are countless comments from regular users reporting the same behavior.
- They are allowed to scrape everybody else but get their feelings hurt when they get scraped....oh yea they respect robots.txt. Guess what, scraping everything that is public is legal.by mrkramer
- It's sad that instead of searching things other people put up, we're basically asking sam or dario oracle to tell us the truth. The people should be furious. But we've internalized this idea that they are somehow better.by Avicebron
- Not better, just much _much_ more convenient. And it doesn't have to be a frontier lab. I'm happy to ask any oracle that returns sufficiently good results which is an ever increasing number of them.by fc417fc802
- At this point, we can all just drop SEO [0]. We are writing content for a robot that hides the source of information.by firefoxd
- but you can click the link and reveal the information. what is being hidden? from who?by adtac
- Wait, so what about the keywords in my meta tags?by user3939382
- We do GEO now and it's not to make the robot link to us, it's to manipulate what the robot thinks.by mitxela
- So why are we angry about that ? I mean the end result for the users are exactly the same, it matters only for bots.
Google have such a (justified) bad reputation that whatever they do, people assume it’s entishification. I don’t believed it is on that matter.
- Aside from the privacy concerns and inconveniences this brings to Google users, it matters for users (of services) that rely on SerpApi too. Kagi for example has started showing these goto URLs in search results.by bramhaag
- For me it's annoying that I cannot right-click the link and copy the actual target. They are breaking the web.by yreg
- Personally, I put a lot of effort into writing my blog posts, some of which have been really well viewed by humans. While mine is effectively a "hobbyist" site, it's nice when readers look at other blog posts as a result of reading the original one. Google's approach ensures that readers get a very narrow view of my content, so everybody expect Google loses out.by FerretFred
- Aside from the list of current concerns that others have covered well...Google is very good at "boiling frogs". That is, rolling out unpopular things in phases.
This could be, for example, step one in the return of AMP, but with a new twist. Where google conveniently returns the content of your website, without directly sending the user to your website. With whatever changes it chooses to make.
by tyingq - Privacy-wise, it enables them to track the result you click on.
Also, though more niche, it would make archived search result pages (e.g. on the Wayback Machine) less useful.
by jolmg - 1. It adds extra latency due to an extra hop.
2. It is non-bypassable tracking of every single click.
3. It does not allow you to inspect the URL to see where it goes to before visiting.
4. It breaks the feedback signal of extensions which redirect sites. E.g. Fandom wikis are shit so I have them redirected to equivalent much better wikis like. But now any such redirection has to be done post-click tracking meaning Google still believes I want to see the Fandom site.
5. It breaks extensions which hide certain shit search results based on URL.
by orlp - It breaks the social contract of the web. My user agent should be able to tell me where a hyperlink goes without first clicking on it, and perhaps act differently based on that information.
Now, all Google search results appear to go back to Google.
by LeoPanthera - It's rather frustrating that after they crawled my website to use for their AI without my consent, and using that AI to slowly strangle my traffic, they're not happy that _other people_ are doing the same to them.by nicbou
- I've been using DuckDuckGo for years now, ever since it became noticeable that two different people searching for the same search term would get two different results back from Google. Meaning they were no longer completely reliable: they might show one person a result that they hide from the other person by burying it on page 3 where few people ever look.
DDG's search results have been poorer recently than they used to — I often see completely unrelated results (to the point of my saying "Why in the world did that come back as a search result??!?") starting from page 2. And yet, I still use them, simply because they aren't Google.
by rmunn - Personalized search results are not ipso facto unreliable.by golly_ned
- 2. They are MS Bing, how is that better?by eviks
- I’ve been using DDG for a long time now. Results are getting more and more spammy. Way too many AI-generated / scraped websites nowadays.by WA
- The age of internet search is over. The age of Cloudflare has begun. It wouldn't be possible to build a search engine now if they wanted to... and there wouldn't be anything to search for anyway. The non-corporate internet withered into dust and blew away in the wind.
If you could find what you want, how would they ever sell you what they want you to buy? And I'm not just talking merchandise, though that too. Your political narratives, your values, opinions, everything. And everyone likes it so much they just sit there scrolling and swiping and tapping.
- I don't know how global consistency is actually useful, but I can easily think of ways it is less convenient.
For example, DST starts and ends on different dates in the UK and the US. Which dates should google return when someone from either country searches for just "DST end date?" Someone lives in Orange County and searches for "Orange County Sheriffs," which one of the eight Orange Counties should google return?
These are both examples where a localized — not even customized, just localized based on IP addresses — search results will easily help reduce headaches.
by pibaker - Speaking of page 3, Google's been asking me to sign in if I try to go beyond page 2 of search results. What a wonderful modern WWW we live in.by _jackdk_
- When Google stopped paid API search a few months ago, I looked for an alternative for my agents that I felt would be sustainable (one-time setup, then out of my mind). I quickly excluded SERP as I feard Google would pull exactly this type of shenanigans to cut them off.
I somehow found Mojeek and settled on it. I had never heard of them. Unlike Kagi, their business model is ads (so they hold no particular moral high ground). But they have a cheap, working paid API.
What I was astonished by is the quality of the results. For my uses, it's undistinguishable from Google. The conventional wisdom is that web search is a Google-sized problem. How did those obscure Brits pull it off?
by stncls - I’m guessing that the knowledge of the techniques to support large-scale web search have diffused out of Google - it has been a few decades after all. Not to mention that the distributed system knowledge that used to live only in Google was either published by Google or cloned in other projects, usually made by ex-Googlers. And you can now rent capacity at scales that 20 years ago required Google to build lots of their own data centers.
Also, there is now a use case for paid search APIs - LLMs and agents - that essentially didn’t exist a few years ago. Not sure why Google hasn’t leaned more into this, but probably a combination of Gemini and Ads interests have combined to view their search index as an increasingly valuable asset, when if anything it might be corrupted already by these interests and therefore be less valuable.
by telotortium - While a lot of people are concerned with local model performance, I wonder how feasible is it now to run a local indexed web search? Surely running an old school Google is possible with the beefy AI rigs today. I know the problem will be crawling which would be bottlenecked by the ISP but I use Google to search SO, Wikipedia, programming language docs, Github issues, and AWS docs. I think a feasible workflow would be to build a set of sites of most interest to you and then prioritize those in crawling.
While typing this out I remembered https://en.wikipedia.org/wiki/Google_Search_Appliance which I never personally used but shows feasibility for the idea. I'm pretty sure one of the newly-announced Macbooks is more than up to the task of matching GSA's offering.
- Plausible. Figure 100 GB each of search index for Stack Overflow, Wikipedia, and GitHub issues, then add a dozen more for docs of all your favorite techs. So maybe half a terabyte. Download and build updated dumps of those once every week or two, and it'd work pretty well. Impractical, but possible.
- Until 10 years ago, i used Dash for that. It’s still around https://kapeli.com/dash
It’s instant, works offline, auto-updates, and includes all the websites you listed, and allows for custom ones too.
by Jakob