Git at Any Scale

Join the discussion

• Hacker News
• Fantastic write up, but I still have doubts. As the author states: Serving git at scale is hard. I can only concur.

  I feel like the article is a bit light on the design of the WAL and maintenance operations, but maybe it's me.

  Anyway, I'd love to have a friendly chat with the author.

  by tex0
• Very cool write up. But I do have to wonder, is Git the right solution for version control given where we are heading? The commands are all muscle memory for me now at this point, but all the projects I've been working on lately have just all moved to trunk based development, rebasing, and squashing merging feature branches onto main.

  I've been working on another project where I have binary files checked in, assets that move or are moved and then changed that git "loses track of" due to the significant change. This just may be me doing silly things like moving a file and then editing it and then committing it and git thinks I deleted it and created a new file.

  by warmwaffles
• That was a very well written and interesting article. I really enjoyed it. Some good background for people telling GitHub to "just scale".
  by dewey
• GitHub issues are mostly with PRs, CI, etc, not with git
  by dgellow
• The problem is that Musk acquired Cursor and therefore Cursor has zero credibility.

  I don't care about what they say because of who they sold out to

  by ljm
• Because politics?
  by cryptonector
• The key part: they decided to use the unchanged git codebase as their building block. Likely assuming that the people who wrote the Linux kernel probably knew what they were doing. That explains some features that look bizarre from the generic distributed-database perspective. I think the author has had this argument many times, because he reiterates it several times in the post. The rest follows from it.
  by gritzko
• This was what stood out to me, too! And this:

  > Packfiles are the fundamental building block of Git storage and Git networking.

  (emphasis mine)

  > His approach was storing the objects in a distributed hash table. This was only possible thanks to JGit, a custom Git implementation in Java. Like any good ol' Java library, JGit provides enough interfaces and factories and interface factories to abstract all the details of a normal Git repository, including replacing its on-disk packfiles with a DHT. Although the system worked and results were good enough for normal Git operations, the limitations of the Git protocol (which again, require packfiles to be sent over the network regardless of how you store data on the server) made the git clone performance bad enough to discard the design altogether.

  Looks like Java enterprise design patterns aren't all bad after all :-) and the git ecosystem would have profited from a bit of abstraction and separation of concerns here, where network protocol, git domain model, and storage layer are evolved somewhat independently. The domain model is what everyone in the ecosystem needs to agree on, the network protocol is what at least the given local & remote host need to agree on, but storage is mainly a local concern.

  Of course the question is whether git would have today's market and mind share if they had gone down that path. The ecosystem would be a lot more heterogeneous, evolving network protocol would probably take much longer, etc.

  by codethief
• It wasn't that long ago that S3 didn't have read-after-write consistency. Crazy to see the explosion in cool stuff built on top after that change. This is a really well written article and definitely changes my perception about Cursor's capability to host a reliable Github replacement. But I could never bring myself to use it because of whose leadership they're under now. It's such a shame.
  by bilalq
• I have to say, when I first heard about Origin, I was skeptical. But now I do believe that the team behind it is capable of pulling it off and shipping something truly incredible.

  This is one of the best technical articles I’ve read. And I came away a lot more knowledgable about a technology I have used daily for over a decade and consider myself adept at. Kudos to the author.

  by iamandoni
• https://cursor.com/en-US/blog/git-at-any-scale to avoid automatic translations (couldn't find a button to switch)
  by wibr
• The language selection is in the footer. Interestingly, the URL for English is the original post without en-US.
  by kibae
• Thank you so much, what an obnoxious behavior!
  by dgellow
• There’s a trend of doing impressive things by pushing many of the hard problems into S3 and assuming S3 “just works”, then not bothering to explain how S3 works. I guess we do the same thing all the time with other miraculous solutions to hard problems, like assuming that superscalar out-of-order processors “just work”, but in cases like this it makes for an unsatisfying explanation. Especially because S3 is a proprietary product, not an algorithm.
  by wrs
• at this point, s3 just means object storage and doesnt mean it actually has to be hosted on aws. theres plenty of other companies that provide s3 compatible storage apis.
  by anthonypasq
• Are there any in-depth technical write-ups on S3 that you would recommend reading on S3 to anyone who _does_ want to get into that abstraction?
  by abhijeetpbodas
• This was the thing that caught my attention. It does feel like a lot of "scaling git was hard and complicated, so we're just offloading a significant to a different layer"

  I've got no dog in this fight and no specific rationale, but it smells like a shortcut that's going to make some assumptions that might surprise later (maybe not at the global level, but a local level)

  by altcognito
• The product is proprietary, but the interface and semantics are documented and freely available. "S3-compatible object store" is a commodity at this point, so it makes sense to treat it as a primitive building block for a distributed system. Of course, each implementation will have its own set of idiosyncrasies and limitations, so you'll still have to make sure that any such divergent behavior doesn't interfere with your own system requirements.
  by 10000truths
• There's nothing Cursor can do that GitHub/Microsoft can't in 2026... And vice versa... after several years of Cursor vibecoding a GitHub clone while catching up with GitHub's new features. Git's performance it's not the issue; it's GitHub Actions, PRs, etc. The moment Cursors steals paid GitHub customers and gets the same workloads, they will start having similar issues! Stealing engineers who couldn't fix this at GitHub won't make much of a difference. GitHub is not just source control; everybody can set up Gitolite and have an amazingly configurable and secure Git setup. I did this 10 years ago, and CODEOWNERS, protected branches, and tags can't do 1% of what I had done before. Cursor is going all over the place as it can't compete with their model. So, this is a defeat, a pivot, not something glorious!
  by nikolay
• Perhaps you didn't read the article, but the implementation they describe is interesting and perhaps easier implemented in a new project from scratch.

  It can't be any worse than current github, and since we've heard plenty of people express their dissatisfaction with github recently, this is actually a good direction for Cursor. A product people may want. We'll see.

  by deburo
• I'd argue that for whatever other faults they have, Composer 2.5 is pretty competitive as an implement-planned-work model. It's several times faster than Sonnet 5, cheaper, and performance is comparable.

  Ears wide open for a model that does better for the same parameters. Pareto efficiency is important, but Anthropic doesn't seem to care about it. They desperately need a Haiku 5, IMO.

  by bric-a-brac
• > What about consensus? Elections? Which server is the primary for a given repository? It also doesn't matter! There's no state and no consensus here. Any server can be the primary. All updates to the write-ahead log are synchronized with an atomic compare-and-swap (CAS) operation on S3, so it's always safe for any instance of a repository to receive a push.

  Again reminded of what an amaizing piece of engineering S3 is (99.999999999% - 11 nines of durability) [1]

  1: https://docs.aws.amazon.com/AmazonS3/latest/userguide/DataDu...

  by biwills
• It is amazing and keeps adding new features that make it better. The CAS stuff is absolutely wonderful and only a recent addition!
  by viccis
• I really enjoyed the article. What's great about the design outlined in it is that it builds on things that work, like leveraging S3 rather than improvising your own distributed storage system.

  Of course S3 goes beyond just Amazon these days. It's a proven design and architecture for doing object storage that you find across different cloud providers that you can also self host if you need to.

  This article will no doubt inspire people working on projects like forgejo, gitea, tangled, etc. that are all facing the same scaling challenges.

  by jillesvangurp
• Not just durability. But read after write guarantee on a distributed object store.

  How many startups and corporations built on top of S3 semantics?

  Snowflake and the big lakehouses are one of them. They use S3 as giant disk in the sky.

  by nojvek
• It’s hard to overstate the reputation of the author of this post. Everything good about GitHub’s internal systems seemed to have his name all over it (I realize that today this statement hits different than a few years ago). Our times at GH didn’t overlap much but hearing the fact that he’s working at cursor increases my estimation of their engineering org by leaps and bounds.
  by brasic
• If he has such a good reputation, why doesn't he write well? He never actually introduces the point of the article, he merely alludes to it in the concluding paragraphs.
  by actinium226
• thanks for the comment ! will read that now

  however whether or not this improves their team - who knows! team dynamics is a hard thing to achieve, not just throwing good people in the same bucket ..

  by realaaa
• Author or not - I just get a shitty machine translation of the original post into German. This is so annoying, machine translated stuff is much harder to read than the original English.

  Oh, and yes the preferred language in my browser is set to English.

  by froh42
• Came here to share a related fact :)

  This solution with WAL and compact resembled the database internals as taught by Martin Kleppmann.

  I asked Claude about this observation, and learned that the author (Vicent Martí) wasn't only core Github developer but also worked on Vitess from PlanetScale.

  So no surprise these two contexts resulted in the design of Continuity as it is.

  https://vitess.io/ - Vitess is a database clustering system for horizontal scaling of MySQL.

  by vindex10