Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- What's up with the images that are supposed to appear in the article? They appear to be coded to load from "./images/containerd-high-disk-io-iotop.png", but https://sealos.io/blog/images/containerd-high-disk-io-iotop.... and https://sealos.io/images/containerd-high-disk-io-iotop.png both fail.
(And indeed, the images are broken in Firefox and Edge. Is there another browser where they're not broken?)
by thaumasiotes - Our platform is designed to solve a very specific workflow, and the DevBox is only the first step in that process.
Our users need to connect their local VS Code, Cursor, or JetBrains IDEs to the cloud environment. The industry-standard extensions for this only speak the SSH protocol. So, to give our users the tools they love, the container must run an SSHD to act as the host.
We aren't just a CDE like Coder or Codespaces. We're trying to provide a fully integrated, end-to-end application lifecycle in one place.
The idea is that a developer on Sealos can:
1. Spin up their DevBox instantly. 2. Code and test their feature in that environment (using their local IDE). 3. Then, from that same platform, package their application into a production-ready, versioned image. 4. And finally, deploy that image directly to a production Kubernetes environment with one click.
That "release" feature was how we let a developer "snapshot" their entire working environment into a deployable image without ever having to write a Dockerfile.
by untrimmed - Who are you, exactly? There is practically no publicly available information about your company, other than that it appears to be held by a Chinese entity called Labring.by otterley
- The irony is that Kubernetes already provides a "ssh into any container" ability, and it's provided directly by k8s, no sshd needed (it's not the ssh protocol but it's good enough to get a shell). Not sure it's advisable to do with any user but an admin, but the standard workflow with k8s is not to shell into running containers anyway, it's to rebuild the container and redeploy the pod.by chuckadams
- Agree with other commenters that this seems like a bad idea. Why on earth should the release image contain all of the cruft of development?? Why on earth should it contain historical versions of all that cruft??by topaz0
- Fascinating deep dive into OverlayFS CoW behavior. The 11GB btmp file getting copied 271 times is a perfect storm scenario. Did they consider mounting /var/log outside the image layers? Seems like that would prevent any log file from causing this amplification. Also interested in image-manip... Does it handle metadata differently than docker export/import?by SurceBeats
- Is it fascinating?
Do people not know that each layer comes with its own downsides?
Do people just do 272 layers and think that it’s normal?
This seems like people discovering that water is wet and fire is hot.
by hug - This is less of a deep dive and more an illustration of the worst way to use containers.
Having /var/log set as as a persistent volume would have worked, but ultimately they were using "docker commit" to amend/update their images which is definitely the wrong way to do it.
by ndsipa_pomu - > image-manip squash: This is the key to reclaiming disk space and the core of our strategy to squash the image layers. The tool creates a temporary container, applies all 272 layers in sequence to an empty root filesystem, and then exports the final, merged filesystem as a single new layer. This flattens the image's bloated history into a lean, optimized final state.
Wouldn't a multistage Dockerfile have accomplished the same thing? smth like
FROM bigimage
RUN rm bigfile
FROM scratch
COPY --from=0 / /
by gkfasdfasdf - I think yep, pretty much. Maybe they didn't know this existed?
- 272 layers in a single image seems really unusual, is that just due to my lack of experience with containers? I've never seen an image with more than maybe a few dozen in my career...by zatkin
- You can build docker images with nix, in which case you can have every dependency be its own layer.
That is clearly not what these people are doing, though.
by paholg - I can't think of anything that would justify that many layers. If I have that much complexity, I would split up the container or start writing bash scripts.
The automation of containers looks simple but developers with systems experience know the actual complexity of operating systems and running applications.
People who know javascript but don't know how a file system works can build and deploy containers. They just copy and paste stuff until it runs. The automation of containers makes brute force iteration a viable option. It was a lot more difficult trying to run a Linux server, which would force you to learn something or use a platform as a service instead.
by kayodelycaon - Well, as described...
> Here's how the disaster unfolded:
> 1. A user's container is under a brute-force attack, and /var/log/btmp grows to 11GB.
> 2. The user performs a commit, creating a new image layer.
> 3. A single new failed login is appended to /var/log/btmp.
> 4. Because of CoW, OverlayFS doesn't just write the new line. It copies the entire 11GB file into the new, upper layer.
> 5. This process repeated 271 times.
So the user is creating hundreds of layers for unclear reasons. The article refers to this as "exponential growth", but for that to be the case those commits would need to be triggered in proportion to the number of existing layers, which seems unlikely. Assuming the commits are caused by the user for reasons unrelated to the size of the existing image, this is growth that is quadratic† (in the number of layers; it's hard to characterize as a function of time or whatever), and it'd be nice to know why there were so many layers.
† Note that while the growth is technically quadratic, I don't think that impacted them. They say that the problem occurred when one 11GB file got copied into each of 272 image layers. That would require 2,992 GB, but they also say that the image exhibiting this problem was only 800GB.
I suspect that the answer here is that only some of the layers modified (and therefore copied) the log file. Probably about 72 of the layers. This is more like growth that's linear (still technically slightly superlinear, but probably not quadratic) in the number of failed SSH login attempts. ~75% of layers aren't contributing to the problem at all.
by thaumasiotes - TIL of `docker commit`. What is the use case for this command? Quick debugging or something, to share with a coworker?by cannonpalms
- Same as snapshotting a vm, or as an interactive version of "docker build". But rarely useful, since most workflows don't really need statefulness.
- In the comments: People who didn't read the article assuming they were literally building 800GB images (the example in the article is an 11GB image that was amplified by copying behaviors)
- >the example in the article is an 11GB image that was amplified by copying behaviors
It's not, it's an 800GB image caused by multiple full writes of an 11GB file into the image's layers. I read the article.
by mejari - The TLDR: > We tackled critical container image bloat on our Sealos platform, fixing a severe disk space exhaustion issue by shrinking an 800GB, 272-layer image to just 2.05GB.
They say they made a 800GB container image, so your issue is about singular vs plural?
Regardless, I don't really get why anyone would self report like this. Is next article going to be about how they don't encrypt passwords and when they accidentally dropped prod DB they could restore account from logs because it had the passwords in clear text?
by guywhocodes - In fairness the article is LLM vomit and could be two paragraphs, can't blame people for not reading it.by progbits
- This whole article could have been much better written as: learn to build images with a Dockerfile/ Containerfile or similar tooling, and store logs in a volume rather than the image filesystem. Everyone that builds a process around `docker commit` is simply in a race against time before they learn this lesson.by bmitch3020
- This seems very much like a ‘we mis configured our containers; then we realised, then we fixed it, then we blogged about it’ post of very little value.
- Right? Blog posts like these makes me question competence instead of attributing it.
- Interesting, although something about the language makes me think it was written by a LLM; I like the ending though:
"The key insight is to treat container images not as opaque black boxes, but as structured, manipulable archives. Deeply understanding the underlying technology, like the OCI image specification, allows for advanced optimization and troubleshooting that goes far beyond standard tooling. This knowledge is essential for preventing issues like Kubernetes disk space exhaustion before they start."
by BinaryIgor - Not a guaranteed tell but I noticed the word "surgically" in the opening paragraphs, and from personal experience I find that appears in ChatGPT a lot for me.
One of the common phrase tropes I find is something like "Here's a set of small, surgical steps you can take to..."
by sunrunner - I’m shocked that a company would share how amazingly bad their layer management had become. This may be a great internal blog, but I wouldn’t share it publicly.by cebert
- From what I understood they provide a kind of shared platform where anyone can run things, and it was one of their clients/users performing the commits.by sally_glance
- Transparency breeds trust.
Sure, it frightens away the short-sighted or particularly excitable people, but anyone who understands how unrealistic perfection is will be comforted by such transparency. Exposing the warts not only sets expectations, but it also assures people that things will (likely) not be just swept under the rug in a company culture of denialism and obfuscation.
by rekabis - It's like a car repair company sharing how they dramatically improved ride comfort, speed and fuel usage by using air to fill tyres rather than concrete.by ndsipa_pomu
- I'm confused. I had the same initial reaction as you and then read further and it sounds like the image was actually provided by a client?
> The problematic user image had an astonishing 272 layers, each representing a commit operation.by cedws