Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- Of course, anything deployed for use by the target customer is a production system. And anything that is under development is a dev system. Pipeline itself is not under development and it is in use by the target customer (developer). What's the confusion here?by zkmon
- > What's the confusion here?
Acknowledging that developers are in fact customers, is often where it comes undone in my experience.
In a lot of places, only revenue paying people are considered customers, everyone else is a cost centre.
Dev tools are often seen as a tax the business pays to be able to generate revenue. Investing in them, or treating them like the production level systems they are, isn’t intuitive for most, and for some a hard cost to justify.
by shimms - Once you enumerate what your development pipeline actually depends on, you will see it rely on a lot of stuff. A typical CI run pulls from npm, PyPI, Docker Hub, a distro mirror and a handful of third-party, none under your control, none with an SLA to you. Third-party outages, yanked versions, compromised packages are the things that can waste your time. Some packages may also drop support for whatever distro you're still using, and the mirrors may disappear (e.g. older Python versions, ROS, etc.).
Shameless plug: To fix all those development pipeline issues, I am working on StableBuild, a tool to easily freeze and pin Docker images, operating system packages, Python packages, and arbitrary build dependencies; in 5 lines of code: https://stablebuild.com .
by malikolivier - The fix is easy and ubiquitous. You host your own artifact repository and docker registry. This also makes sure your supply chain remains easily auditable.by rcbdev
- > CI run pulls from npm, PyPI, Docker Hub, a distro mirror and a handful of third-party, none under your control, none with an SLA to you.
To be honest, they shouldn't do so directly. For the obvious reason you stated, they don't have a SLA to you or your company. But also because it is a shitty thing to do as a company. Companies should have their own proxies and mirrors for any repository they pull from. Both to be in control of the dependencies you as a company rely on and to not unnecessarily put load on free third party repositories.
Had not heard of your solution. More often than not I come across Harbor, Nexus or things like GCPs internal stuff.
by creesch - I guess the problem is visibility. In the same way that many companies don't take security seriously until it becomes a problem. If a problem never manifests, but was prevented, was it really a problem?
Of course I know the importance of preventing problems, but how do you properly measure it? A data leak is immediately visible, but preventing them is less so.
by chrysoprace - by esafak
- > how do you properly measure it? > If a problem never manifests, but was prevented, was it really a problem?
For the scheduled half of it, i like inverting the alert. Instead of alerting when a job fails or wait for data leak, have the job check in on every successful run and alert when a check-in doesn't arrive. Dead man's switch. Ten minutes to set up, and it's the difference between "the cron is fine" and a statement that can actually turn out to be false.
Agree that it is difficult to implement for any process, but it is easier to create validation systems than "find the error" processes.
by adoltech - No it's not. If the development pipeline breaks on Saturday night there is absolutely no need to go fix it right away.
You should have a method to deploy hotfixes to code that bypasses your development pipeline though, in case the development pipeline breaks while you need to fix production- because you must be able to fix PRODUCTION even if your dev pipeline is broken ( because again your dev pipeline, while important, is not production)
by Illniyar - I feel like your example exactly describes why the development pipeline is production. Just because the critical hours of the system are different doesn't mean it's any less important.by dylanowen
- Worth mentioning CI is not part of production outages!
You should never fix an incident by deploying another fix as it risks making things worse especially while everyone is panicking. Better to roll back - and spread non-backwards compatible work across multiple deploys that each are backwards compatible - always giving yourself a known good state to roll back to.
Rollback should not done be via your CI either! Set up some low dependency thing - a separate tool or script or system.
Once you've rolled back and got the system stable, then produced a fix at your leisure, that's when CI comes back into the picture :)
by cadamsdotcom - Having spent too many years in CI/CD I'd agree but there is an even better solution; decouple deployments from releases. Make it possible to deploy without releasing, and release without deploying. (An implementation of this could be, for example: feature flags.)
- I’d bet there are as many stories of businesses failing because of inability to ship quickly as there are about focusing too much on your tooling instead of delivering value to customers. Both failures are dangerous! And depending on your market, product, team etc. a different spot on that spectrum is appropriate.
In some places shipping quickly is part of the value. In others, the product works and delivers value and having slow releases (implying eg a manual release process) is a feature.
What this article advocates for is certainly a valid lens, by IMO it’d be a mistake to take it as universal.
by maciejgryka - I worked at a startup that was briefly a unicorn and ended up burning cash because they did exactly what you said of focusing a lot on tooling and dev environment. It was great but way over the top for what we needed. I learned a lot of good practices for myself but they've been limping along for a while and most of my former team was laid off or saw what was coming and left before it hit them.by ctkhn
- So is the recruiting pipeline. Kill that and you kill the company. Agents and AI can only do so much, the rest is people power. Sadly too many companies have broken recruiting processes.by reactordev
- That sounds nice and pleasantly confirms our biases, but it's blatantly false.
Existence of recruiters and recruitment companies as separate job entities is a proof positive that you do not, in fact, need a working recruitment pipeline.
If anything, recruitment in our industry was always broken. And yet here we are.
by TeMPOraL - There's more to making an argument than stating your conclusion forcefully and repeatedly.
EDIT. In particular the author hasn't acknowledged the prioritization problem, hasn't explained what factors might go into the necessary cost benefit analysis. The author merely points out "developer productivity is important!" You might as well say "invest in your tools!"
by dsjoerg - Not seeing this get traction, but this is exactly what I thought while reading this. The problem with calling something "a" top priority is you can only have one top priority. That's how priority works. A customer facing outage is a higher priority than a company-wide developer tool outage, which is higher priority than a single developer's personal workstation failing, etc. etc. That doesn't mean the lower-priority things don't matter, but you can't make any and every failure of anything an all hands on deck situation. That's a certain path to burning out your entire company.by nonameiguess
- This is the first thing you learn when you move into DevOps. And something management don’t appreciate when DevOps engineers ask for another environment just for themselves (ie somewhere they can dev infra without breaking stuff for developers who are currently building stuff in dev)by hnlmorg
- Yeah at my place our DevOps team had been asking for resources on our CI/CD pipeline for a long time but wasn't prioritized. After awhile, after introducing something like teams wide "Retros', we noticed they didnt want the resources, we wanted them!
So we started being more vocal and management understood its for everyone and their wish got approved recently and its been a big change.
by 1dontnkow_ - When we do updates, I mandate that our teams do blue/green deployments meaning they setup the exact VM side by side with the actual production data and test with the new version of software (eg GitLab) before doing the DNS flip. This is on prem. It’s an ephemeral sandbox.
- > If the QA server is down, the testers are unable to do their jobs, and the team isn’t producing working software. For the QA team, this is a production outage. Fixing it should be a top priority.
Genuine question, does anyone here ITT working in software still have dedicated QA? They laid off all our QA engineers about a year ago, and talking to friends and former colleagues it seems to be the industry wide trend?
FWW I think a good QA person is worth their weight in gold and this has been a horrible mistake. I'm just curious if there are any left.
by donatj - Different country probably, here QA is still often part of the team. More often than not as part of a team with a title like "test automation engineer".by creesch
- Worked for a company with a 3D web application (among other things.)
Those things can't be tested with Playwright et al. So the team had a "every pull request gets an instance like pr143.company.org" pipeline where QA approved. They must still have it.
The other companies tested "on demand."
by stephbook - I'm in legacy/finance right now and we have a QA team that we share with maybe 2 or 3 other dev teams. I think our integration testing systems could be a lot more efficient but I think if we had a proper pipeline that truly guaranteed integration testing (right now it feels like a shrug) we would lose the QA team like you did.by ctkhn
- Here in Finland my company has a dedicated testing team that's almost at large at the actual engineering staff. We sell a combined hardware-software stack, though, and it's basically a contractual requirement in our industry. They are very helpful to be on good terms with when working on a given project, they often know ways to work with the gestalt I never would have guessed in a million years.
- Yes. We call them BAs and they have deep product knowledge, so they also get used for training, escalated support requests, estimations on effort required in new integrations, etc.by lelanthran
- My experience is 50/50 - half have dropped them, half still have them.
I also don't find most QA to be worthwhile compared to a good, quality focused dev.
There are several cultural issues with how QA are treated (low pay) and how they think they're expected to behave (e.g. hands off the code, hands off types, hands off unit tests) which make them less useful than a dev who is really good at engineering for reliability.
Where we had good engineering practices and that type of "black box" QA I noticed that product tended to find the majority of bugs, followed by devs, followed by customers, followed by QA.
Only when engineering practices sucked and they were used as a crutch by devs did they really catch a lot of bugs.
Of course, in these days of vibe coded madness they're probably getting more useful, not less.
by pydry - We have a QA team but it’s small in comparison to the dev team. We didn’t have one before as we thought automated tests and “good developers” who take testing seriously was enough. Unfortunately, even with extreme automation and huge test suites, our bug count per release was unacceptable. Having a QA team was a big help in fixing that. We have two weeks of testing before a release goes out. But we sell a security product so for us bugs must be far between. If you just deploy web apps and you can update every hour, I guess you don’t need it as much.by brabel
- > Genuine question, does anyone here ITT working in software still have dedicated QA?
A few years ago I worked at a company who had a team of QAs dedicated to do manual acceptance tests. Up until the day where a critical regression went unnoticed and it turned out QAs were going through the test suite without actually checking the behavior. Following that the company got rid of all QAs except the software engineer in testing who was responsible for the whole automated test infrastructure, and instead of QAs the team held test days to go through the same manual test suite while we progressively automated them away.
And the whole process improved all across the board.