Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- > I still don't totally get why the shift happened when it did. Five years ago all three camps were doing fine. Now the VM+systemd crowd has basically disappeared from job postings, serverless stayed niche, and K8s just won. > > My best guesses: managed K8s (EKS, GKE, AKS) got mature and the talent pool flipped: enough people learned it that hiring for anything else became the harder choice. And Helm made "just use someone else's chart" a real option. But I'm not certain. If you were there for the shift and have a better theory, I'd genuinely like to know.
Pretty much, almost. Have spent a bunch of time in my career working on the "VM + systemd" setups, stuff running on a rack, or in an ec2 on cloud - managed kubernetes is a lot better for me than those cobbled together messes. There's "easier" setups but usually end up costing me a lot more in time and $.
To answer simply, it became good + convenient. I could complain about plenty, and people here like to, but honestly you couldn't pay me to go back to the old way. The one legitimate gripe is the upgrade schedule is exhausting, on AWS it's about every 6 months before you go into extended support. I also hate being at the mercy of arbitrary decisions like "ok we know a huge chunk of the web going back a decade has architected off our Ingress API, but recently we decided we dont really like that way anymore and we want you to use Gateway API instead, so, um, like ya we know it just killed off one of the most used open source ingress configs (ingress-nginx) but yea trust us bro this is going to be so much better" kind of thing.
by JohnMakin - What I've seen more than anything else is that Kubernetes built an ecosystem (of contributors and users, but also of companies invested in its success) that none of its competitors could or would. There was apparently a faction within Google that believed open-sourcing Kubernetes was a mistake because Google would have made more money keeping it in-house, but in terms of the success of the project I think it was entirely the right call, as was creating a foundation to maintain and promote it. Look at the history of its competition:
* DC/OS was always its own thing and as time went on, eventually Mesosphere was basically the sole maintainer of the underlying Mesos. Very little external contribution.
* OpenShift was different from Mesos and basically maintained only by Red Hat from the Makara acquisition (sometime in 2010 I think) to about mid-2015 (i.e. the point where they ripped out most of the OpenShift-native process isolation and orchestration and replaced it with Docker and Kubernetes). Pre-Kubernetes OpenShift frankly struggled to catch on and again, basically everybody who cared about developing it worked for one company.
* CoreOS was developing fleet in the open but dropped it outright when Kubernetes was released. The phrase I heard there was "We started to say something and Google finished our sentence." They pivoted to Kubernetes for orchestration so hard it was kind of awkward talking to customers who used fleet after that. In theory somebody could have picked it up like Kinvolk picked up rkt for awhile (and later CoreOS Linux as Flatcar), but as far as I know nobody ever made a serious effort to do so.
* Docker released Docker Swarm shortly after Kubernetes was released -- yet another one-company product. (I still don't really understand why they released Swarm -- for simple workloads, Docker Engine and Docker Compose were enough, and for more complex ones Docker Engine was, at that time, still the sole underlying runtime in Kubernetes. There were already two distinct orchestrators on the market, one from a much larger company with a lot more operational experience running containerized workloads than Docker had. What was their thought process?)
* HashiCorp released Nomad well after Kubernetes but not only was it another sole-corporate-maintainer orchestrator, it deliberately omitted a lot of the basics Kubernetes included like service discovery in an effort to stay simple -- so in very few cases was Nomad alone actually enough to orchestrate workloads (nor was it intended to be, as the Nomad engineers in the ~1.0 days would have been first to tell you). Past a point this made Nomad more work to get running and keep running than Kubernetes was.
The flip side is, I don't think a purely community-developed orchestrator would have won, even with a foundation backing it. It's not the corporate backing that's the issue, it's the lack of diversity in that corporate backing.
by kensey - The upgrade cycle is a feature, not a bug. If (when) you need to do a big lift and shift, or there's some 0 day CVE, push buttan, get security update. You CAN drift behind but there's a real $$$ cost to that now. Every three months I toss opus at my k8s stack and verify it's compliant with k8s v1.xx.y and then push the upgrade button on my staging cluster, and then a week later I push the upgrade button on my prod cluster. What used to be two days of maintenance every quarter is now more like 2-5 minutes spread across the two upgrades.
I'll admit I'm dreading switching over to the gateway api, but by the time I get forced off ingresses it should be a stable/mature ecosystem. That's still a ways out though.
I don't know anyone still dealing with VMs anymore, except our IT guy who manages a couple of pet servers for random executives from the before times. In the last year k8s has started absorbing executive pet processes and the number of VMs our IT guy manages has dropped by about half.
While I'm here spouting stuff, yeah hiring for k8s is real easy, if our SRE gets hit by a bus, he can be replaced in a week, and we can probably struggle through using opus until that happens. K8s being he lingua franca of git ops IaC makes it real easy for the new guy to parachute in and start working. Every VM thing is going to be totally bespoke and have the personality of the guy who designed it, which is rarely a good thing.
by hadlock - I somewhat agree with you... but it's not like you don't need some actual experts who know what they're doing, especially when stuff goes bonkers and it will go bonkers.
Even on AWS EKS, you will run into bullshit with their network overlay. Egress policies are a mess (at least half a year ago, you were not able to say something like "allow pod A to egress traffic to service (!) B" despite a service resolving down to an IP address in the end.
And that's before going into the unholy mess that is getting connectivity to and from the external world to your cluster. Cloudfront, ACM certificates, ALB, ALB-EKS integration, Route53, Route53-EKS integration, EFS, EFS-EKS integration, EBS, EBS-EKS integration, RDS, RDS-EKS integration, IAM-EKS integration, SSM, SSM-EKS integration, autoscaling... and if you want more pain and don't already wince, try setting that up across regions or, as I had to do once, across account boundaries.
Kubernetes is powerful. But do not make the mistake of assuming it's easy to get started with, at least on the admin side. Even if you got prior AWS experience, getting it all integrated into EKS so you don't have to deal with Terraform and helm/k8s for a full deployment of a piece of software will take you an awful lot of time.
For users though? It's a breeze, I will admit as much. Everything down to the firewall rules can be encoded in k8s spec files.
by mschuster91 - Kubernetes is so ubiquitous that yeah, as long as you're not trying to run it yourself, a small Kubernetes cluster just isn't all that much to manage. I think it's been so long, people forget how annoying it is to manage servers. All said and done Kubernetes is becoming more the "Boring Technology"by mikgp
- This seems to be less about K8’s and more about the infra as code movement. It doesn’t matter if you use K8, CDK, or terraform - you get the same benefits the OP stated across the board.
It is nice to be able to have a consistent deployment pattern, with traceability, rollback support, and production approval checks. It’s nice to not have some archaic something stuck in someone’s head. It’s also nice to be able to see how something works by reading the code, which is usually up to date and deployable.
by SamuelAdams - > less about K8’s and more about the infra as code movement. It doesn’t matter if you use K8, CDK, or terraform - you get the same benefits the OP stated
I’d like to gently push back on that. ;-D
Terraform, when committed to git, provides organisational memory. But less so uniformity, since all providers are different (and you should expect different things when applying). No tracing besides git. And tfstate is hard to share between developers, unlike kube state.
Kubernetes is more the same across providers. And it manages drift after something is applied, which is not a direct argument of OP, but a strong reason over other IAC.
And yes, I also enjoy how well deploying works. And how things generally fit together. Liking the networking complexity less so.
by sshine - > The CTOs I talked to aren't making a dumb choice. They're solving real problems.
Unrelated to the content of the article, this sentence structure is a dead giveaway of LLM writing.
by shevis - This blurb gave me the idea to try and quantize this. Scrape the top HN blogs over the last few years and see how occurences of common phrases change.
I'd expect to see a huge increase in "solving real problems" over the last months.
by jcattle - It's not uniformity, it's cargo culting and offloading thinking to group norms. Doesn't help engineers are some of the most arrogant people alive and refuse to admit anything is complicated, as they consider it some kind affront to their intelligence.
I would not advise asking the majority of CTOs these questions either. Many got to that position by saying what people want to hear, which is the "average" safe answer. They will parrot whatever is "hot" at that time because it's the least risky response. They are not your friend nor a reliable source.
by codemog - I think a big part of this whole discussion (and why it's always such a divisive topic) is that there are so many factors to consider that there is no one single golden bullet. An industry standard just makes this issue go away; also it makes the decision making easier since you're not taking risks going for "VM-with-systemd" or "plain docker" or "bare metal" over $STANDARD.
> I would not advise asking the majority of CTOs these questions either. Many got to that position by saying what people want to hear, which is the "average" safe answer.
Agree; this is the same as asking people why they're not having kids: they either a) don't know or b) don't want to / are not willing to say the truth.
by quibono - Well, I totally get the benefits that made those people choose Kubernetes. It's just that those benefits could be had w/o running a massively complicated piece of machinery that is mostly engineered to solve problems I don't have.by avhception
- Adopting k8s when you hire your _second_ engineer (first after the CTO)? That’s a red flag that the CTO’s priorities are wrong and he’s just enjoying tinkering with his infra instead of solving the users’ problems.
- I thought that was the point of the article, right?
That the tech benefits may not be there, but they’re using it for the non-tech benefits
by Esophagus4 - None of those arguments make sense.
Uniformity ? Try deploying openbao inside kube, if kube decides to restart your pods, you're in for unsealing them at 3am, waking up everybody who owns a Shamir key. So bao stays out of the cluster, or pinned to certain nodes, defeating the purpose entirely. Also, with the ultra wide variety of tools at every layer of the stack, uniformity is a joke ; there are no 2 kube cluster deployment that are the same really.
Standardized knowledge ? The operating system is standardized knowledge. Any competent SRE should be able to login into a Linux box and figure out what's running there. And if you let your previous ops shadow it all you're just a pretty bad CTO.
Tracing who does what ? First of all anybody with admin access can run one time jobs just like anybody with sudo can run one time commands. That's like chapter 01 of the kube doc. Also again at the kube layer itself, below the helm chart, the ops who set that up or updates it can and will change stuff that breaks stuff.
Kube isn't necessarily bad and has it's purpose but it's not a product. It's like Linux, a complex piece of tech that requires a lot more knowledge than "just push this helm chart" to work.
by bionsystem - Sounds like you had specific issues with openbap and your cluster provider. Whatever the tech stack it's all presented the same way and easily discoverable. Kubernetes is a cloud operating system- this is exactly the point about standardized knowledge.by bakies
- There is a core 20% to Kubernetes which is very nice, mostly being the Deployment and Service management stuff. That along with a very basic GitOps for cluster management (an infra repo for operators using Flux, applying service level yaml from app repos in CI) above a cloud managed Kubernetes cluster, where you still keep your DB and build servers off the cluster, can be quite nice for a small team.
Beyond that, there are massive holes of despair to fall down if a novice team starts to engage with extensive operators (starving the control plane), DB operators (distributed persistence) and build operators (spikey, expensive loads). At least, I know that I've had to dig out of those holes.
I just hope people don't use k8s in the same way many use microservices: as a way to introduce complexity for complexity's sake.
by liampulles - Spot on. I have a lot of trouble convincing cloud folks that for durable state, you probably don’t want kubernetes. It’s not that e.g. the CSI drivers and operators for clustered databases aren’t top notch—they are; the era of “avoid stateful kube services” is long behind us—it’s that the cloud provider managed services for e.g. blob stores or databases are so much more reliable. The S3s and Auroras of the world are expensive for a reason: no matter how good your kube native database operator is, it still doesn’t assume responsibility for a ton of the failure points that managed services do. And that’s true even at modest scale (e.g. upgrades are just harder when you’re running your own DB) and in cost conscious environments (sure, the Elasticache bill is steep, but the salary and velocity cost of fixing memory-leak-caused kube memcached crashes is steeper).by zbentley
- So I’m personally a huge fan of k8s and while I agree it may be „complicated”, it’s because deploying applications is complicated. (I want to point out that there is no requirement no set up cert manager, ArgoCD, external secrets, etc. - and many people who’d consider a VPS would happily slap a .env with an unencrypted secret then ssh to update, but when they choose Kubernetes they take the long route of doing proper GitOps and complain that there are so many things to configure :)
But I found funny that the OP summarized to use Kubernetes when CTO is no longer the only dev.
by suralind - > many people who’d consider a VPS would happily slap a .env with an unencrypted secret then ssh to update
I just want to point out that you can totally still do this with Kubernetes. Of course it's not correct, but you can save that unencrypted secret in a .env file right into your container while you're building it - no need to use Kubernetes's support for supplying environment variables from the manifest. And of course, you don't even need a Dockerfile to build that container - you can just exec into a running container, paste it in, and then docker save.
Kubernetes doesn't save you from making stupid decisions, it just makes it easier to make better ones.
by solatic - 100% agree with you.
You can actually treat kubernetes as a glorified docker compose engine. Deploy pods, deploy nginx instead of ingress controller, deploy certbot cronjob instead of cert-manager, and believe it or not, it'll work! On a single server!
People often compare Kubernetes with thousands of additional services to a simple VPS, but that's not apples to apples comparison.
by vbezhenar - One year ago I might agree that Kubernetes is an overkill but today?
Ask your favorite GPT to generate manifests, get primary app into cluster with telepresence or execute straight from container and switch contexts and clusters like it's 90s again.
One reason I dislike Docker Compose and Docker is lack of isolation. Yes sure if you put your arm deep enough you can get it, but on local k8s I can spin cluster per workspace and not worry about conflicting ports between PostgreSQL instances.
Before LLMs writing consistent YAMLs was PITA but today on low/development scale it's pretty much free lunch.
by xlii - Interesting. I have just started reading about Kubernetes. Is there an reading material that goes over this process you just described?by iamcreasy
- > Ask your favorite GPT to generate manifests, ... > Before LLMs writing consistent YAMLs was PITA but today on low/development scale it's pretty much free lunch.
Writing manifests seems like a trivial thing to focus on. Who operates the k8s cluster in production? Who runs upgrades? Who's on call to monitor the system? Of course if someone else is doing all the work for you, it feels like free lunch!
by perrygeo - This cuts both ways.
At any stage of https://www.macchaffee.com/blog/2024/you-have-built-a-kubern... a SOTA model can repackage it into Kubernetes.
If you're feeling extra spicy you don't even need the deploy scripts. Just a `llm` user account with the right permissions & ssh keys on all your servers.
by athrowaway3z - Finally just bought a piece of my own hardware and got LLM to deploy k3s cluster on it.
I think diy homelab/hosting is more accessible than ever.
Cut costs on cloud spend and invest into AI spend.
For a solo dev on a budget, I think it just makes sense.
by aleksiy123 - > One reason I dislike Docker Compose and Docker is lack of isolation. Yes sure if you put your arm deep enough you can get it, but on local k8s I can spin cluster per workspace and not worry about conflicting ports between PostgreSQL instances.
Using Kubernetes because you're unable to grok docker's networking enough so you can't run multiple containers using their own ports and not conflicting with other stuff sounds like a recipe for disaster, even (especially?) if you use agents for this. Particularly if you let them manage a production environment, you're bound to lose important data eventually.
> pretty much free lunch.
Aah, famous last words of the young :)
- Strong agree, if there's one thing LLMs are excellent at, it's writing Terraform and Kubernetes deployments (and/or helm charts). What used to be half a day of research, trial and error, is now 20 seconds of AI churn and 98% of the time it nails it on the first try. And then point it at grafana and tell it to write you a dashboard for the new service/s. Easy peasy lemon squeezy. What used to require a team of 4 devops/SRE to support a medium sized company, can now be collapsed down into a a single part time SRE.by hadlock
- I made this decision at a startup (albeit when the eng team was ~30 people, and we had a monolith with ~10 supporting services). I wouldn’t do it again, even for the reasons stated in the article.
The uniformity is nice, we were moving from apps running directly ec2 instances provisioned with ansible. Each time we spun up a new service it was a process to get the ec2 instances provisioned just so.
But k8s is such a pain in the ass. One thing that I think people new to it don’t realize is that it’s not at all batteries included - to get a basic managed cluster setup, you’re still going to be installing a bunch of additional controllers (ingress, cert-manager, external dns to start). And then you’re on the hook for making sure all those processes stay up (hope the admission webhook controller for a critical resource doesn’t go down!). Then you’ve got to do a major upgrade on not only your cluster, but all of those controllers every ~3 months. And no one is shy about introducing breaking changes.
Also you’re introducing a huge amount of complexity with the k8s networking and dns layer that most startups have zero need for (if you’re on EKS, make sure to read about scaling and monitoring CoreDNS).
I think there is a real hole in the market for a simple solution that lets you deploy some containers to some instances in a declarative fashion without all of that complexity and does decent LTS versions. I imagine there’s something out there that does this, but k8s has really sucked up all the oxygen.
by mikeocool - k8s is not a pain, I would never return to something like Pupet / Ansible / etc ... to deploy bare ec2 instances, it's just re-inventing the wheel badly.
Just use ECS / Fargate with an ALB in front if you need a simpler use case.
by Thaxll