Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- It's an interesting article, thanks for that.
What people forget about the OVH or Hetzner comparison is that for those entry servers they are known for, think the Advance line with OVH or AX with Hetzner. Those boxes come with some drawbacks.
The OVH Advance line for example comes without ECC memory, in a server, that might host databases. It's a disaster waiting to happen. There is no option to add ECC memory with the Advance line, so you have to use Scale or High Grade servers, which are far from "affordable".
Hetzner per default comes with a single PSU, a single uplink. Yes, if nothing happens this is probably fine, but if you need a reliable private network or 10G this will cost extra.
by nik736 - Is there software that works without ECC RAM ? I think most popular databases just assume memory never corrupts .by vjerancrnjak
- Their current advance offerings use AMD EPYC 4004 with on-die ECC. I can’t figure out if it’s “real” single correction double detection, or if the data lines between the processor and dimms are protected or not though.by hedora
- I can't believe how affordable Hetzner is. I just rented a bare metal 48 core AMD EPYC 9454P with 256 GB of ram and two 2 TB NVME ssds for $200/month (or $0.37 per hour). Its hard to directly compare with AWS, but I think its about 10x cheaper.by montecarl
- Yes, but there are options for dedicated server providers who offer dual PSU and ECC ram etc. It's more expensive though for e.g a 24 Core Epyc with 384GB RAM dual 10G netowork is like $500/month (though there's smaller servers on serversearcher.com for other examples)by jammo
- I never understood the draw of 'server-grade hardware'. Consumer hardware fails rarely enough that you could 2x your infra and still be paying less.by torginus
- These concerns are exaggerated. I've been running on Hetzner, OVH and friends for 20 years. During that time I've had only two issues, one about 15 years ago when a PSU failed on one of the servers, and another a few years ago when an OVH data center caught fire and one of the servers went down. There have been no other hardware issues. YMMV.by lossolo
- I had a problem figuring out why the place I was working wanted to move from in-house to AWS; their workload was easily handled by a few servers, they had no big bursts of traffic, and they didn't need any of the specialized features of AWS.
Eventually, I realized that it was because the devs wanted to put "AWS" on their resumes. I wondered how long it would take management to catch on that they were being used as a place to spruce up your resume before moving on to catch bigger fish.
But not long after, I realized that the management was doing the same thing. "Led a team migration to AWS" looked good on their resume, also, and they also intended to move on/up. Shortly after I left, the place got bought and the building it was in is empty now.
I wonder, now that Amazon is having layoffs and Big Tech generally is not as many people's target employer, will "migrated off of AWS to in-house servers" be what devs (and management) want on their resume?
by rossdavidh - with "dev wanting X" nothing happens. "leadership deciding X" then it needs to get done.by ahel
- Devs wanting to put AWS on their resume push for it, then the next wave you hire only knows AWS.
And then discussions on how to move forward are held between people that only know AWS and people who want to use other stuff, but only one side is transparent about it.
by whstl - > Equinix Metal got the closest, but bare metal on-demand still carried a 25-30% premium over our CapEx plan. Their global footprint is tempting; we may still use them for short-lived expansion.
> The Equinix Metal service will be sunset on June 30, 2026.
- “You lean heavily on managed services (Aurora Serverless, Kinesis, Step Functions) where the operational load is the value prop.”
Not viable even when your core belongs to AWS. Why? Ask prime video
- FD: I work at Amazon, I also started my career in a time where I had to submit paper requests for servers that had turn around times measured in months.
I just don't see it. Given the nature of the services they offer it's just too risky not to use as much managed stuff with SLAs as possible. k8s alone is a very complicated control plane + a freaking database that is hard to keep happy if it's not completely static. In a prior life I went very deep on k8s, including self managing clusters and it's just too fragile, I literally had to contribute patches to etcd and I'm not a db engineer. I kept reading the post and seeing future failure point after future failure point.
The other aspect is there doesn't seem to be an honest assessment of the tradeoffs. It's all peaches and cream, no downsides, no tradeoffs, no risk assessment etc.
by yanslookup - Variants like k3 are not as complicated and problematic as k8.by dumbledoren
- I agree that a business should use Kubernetes only if there is a clear need for that level of infrastructure automation. It's a time and money mistake to use K8s by default.
- Indeed Kubernetes is problematic for the complexity and fragility reasons. It's a scalability problem, it's designed for big scale with kube staffs and situations where cost savings from bin packing etc outweigh costs of the resulting complexity.
But, SLAs are no good (who cares about getting refunded).
by fulafel - Managing a complex environment is hard, no matter whether that’s deployed on AWS or on prem. You always need skilled workers. On one platform you need k8s experts. On the other platform you need AWS experts. Let’s not pretend like AWS is a simple one-click fire and forget solution.
And let’s be very real here: if your cloud service goes down for a few hours because you screwed something up, or because AWS deployed some bad DNS rules again, the world moves on. At the end of the day, nobody gives a shit.
by AndroTux - At another big-4 hyperscaler, we ended up with substantial downtime and a lossy migration because they didn’t know how to manage kubernetes.
Microk8s doesn’t use etcd (they have their own, simpler thing), which seems like a good tradeoff at single rack scale: https://benbrougher.tech/posts/microk8s-6-months-later/
The article’s deployment has a spare rack in a second DC and they do a monthly cutover to AWS in case the colo provider has a two site issue.
Spending time on that would make me sleep much better than hardening a deployment of etcd running inside a single point of failure.
What other problems do you see with the article? (Their monthly time estimates seem too low to me - they’re all 10x better than I’ve seen for well-run public cloud infrastructure that is comparable to their setup).
by hedora - Many startups and companies couldn't exist if there was only AWS (or GCP / Azure) due to how much they overcharge.
For example, we couldn't offer free GeoIP downloads[0] if we were charged the outrageous $0.09 / GB, and the same is true for companies serving AI models or game assets.
But what makes me almost sick is how slow is the cloud. From network-attached disks to overcrowded CPUs, everything is so slooooow.
My experience is that the cloud is a good thing between 0-10,000 $ / month. But you should seriously consider renting bare-metal servers or owning your own after that. You can "over-provision" as much as you want when you get 10-20x (real numbers) the performance for 25% of the price.
by pingoo101010 - I’ve seen cloud slowness create weird Stockholm syndrome effects, especially around disk latency.
It always makes sense to compare to back of the envelope bare metal numbers before rearchitecting your stack to work around some dumb cloud performance issue.
by hedora - Many other points. When the Cloud Started, they offered great value in adjacent product and services. Scaling was painful, getting bare metal hardware have long lead time, provisioning takes time. DC was not of as high quality, Network wasn't as redundant. A lot of these today are much less of an issue.
In 2010 you could only get 64 Core Xeon CPU coming in 8 Sockets, or maximum or 8 Core per socket. And that is ignoring NUMA issues. Today you could get 256 Core per socket that is at least twice as fast per core. What used to be 64 Server could now be fitted into 1. And by 2030, it would be closer to 100 to 1 ratio. Not to mention Software on Server has gotten a lot faster compared to 2010. PHP, Python, Ruby, Java, ASP or even Perl. If we added up everything I wouldn't be surprised we are 200 or 300 to 1 ratio compared to 2010.
I am pretty sure there is some version of Oxide in the pipeline that will catch up to latest Zen CPU Core. If a server isn't enough, a few Oxide Rack should fit 99% of Internet companies usage.
by ksec - > Cloud makes sense when elasticity matters; bare metal wins when baseload dominates.
This really is the crux of the matter in my opinion, at least for applications (databases and so on is in my opinion more nuanced). I've only worked at one place where using cloud functions made sense (keeping it somewhat vague here): data ingestion from stations that could be EXTREMELY bursty. Usually we got data from the stations at roughly midnight every day, nothing a regular server couldn't handle, but occasionally a station would come back online after weeks or new stations got connected etc which produced incredible load for a very short amount of time when we fetched, parsed and handled each packet. Instead of queuing things for ages we could instead just horizontally scale it out to handle the pressure.
by sondr3 - The core of this success is this, IMO:
Which TBH applies to many, many places, even if they are not aware of it.> Our workload is 24/7 steady. We were already at >90% reservation coverage; there was no idle burst capacity to “right size” away. If we had the kind of bursty compute profile many commenters referenced, the choice would be different.by darkwater - Even if you have that you'll find AWS is "out of stock" and wants you to create reservations that essentially cost the same as just having the machine 24/7.by jdsully
- YES, but let's also not ignore that the market is cold right now and the tech industry isn't really growing like crazy like it did during ZIRP.
So yeah in a hot economy anything you launch grows crazy. Then you do the thought leader talk circuit.
And in a cold economy, you can stop growing, optimize for your 24/7 steady workload, and also do the thought leader talk circuit.
by deanCommie - Nah. They could have just overprovisioned to hell for much cheaper. Boxes at Hetzner cost up to 10 times less than equal level of AWS compute. Just overprovision for cheaper. You have to overprovision on the cloud anyway - you cant risk your users waiting 1-2 minutes until your new nodes/pods come up. So 'cloud is good for spiky load' argument is just a lie we tell ourselves.by dumbledoren
- Reminds me of https://www.specbranch.com/posts/one-big-server/by Nemo_bis
- I'd say the core of their success is running everything in a single rack in a single datacenter at first (for months? a year?) and getting lucky. Life is simple when you don't need the costs and effort of reliability upfront.by marcinzm
- These are the features that AWS provides
(1) Massive expansion of budget (100 - 1000x) to support empire building. Instead of one minimum-wage sysadmin with 2 high-availability, maxed-out servers for 20K - 40K (and 4-hour response time from Dell/HPE), you can have 100M multi-cloud Kubernetes + Lambda + a mix-and-match of various locked-in cloud services (DB, etc.). And you can have a large army of SRE/DevOps. You get power and influence as a VP of Cloud this and that and 300 - 1000 people reporting to you.
(2) OpEx instead of CapEx
(3) All leaders are completely clueless about hiring the right people in tech. They hire their incompetent buddies who hire their cronies. Data centers can run at scale with 5-10 good people. However, they hire 3000 horrible, incompetent, and toxic people, and they build lots of paperwork, bureaucracy, and approvals around it. Before AWS, it was VMware's internal cloud that ran most companies. Getting bare metal or a VM will take months to years, and many, many meetings and escalations. With AWS, here is my credit card, pls gimme 2 Vms is the biggest feature.
- > one minimum-wage sysadmin
The internet assures me there are loads of these underemployed Unix/networking experts just sitting around waiting to set up your infrastructure. But in my experience, these people are actually really difficult to hire, and not at all cheap. (Possibly the sharp ones have 'sold out' and gone the SRE route and are now one of those '3000' people.)
So I wonder if there's a certain amount of wishful thinking on both sides here, like "I wish a 'clueful' company would hire me to be their head sysadmin...", while companies who have tried to do this on the cheap usually just have terrible ops. ("Whoops, the backups haven't worked in 2 years...")
by Slothrop99