Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- This service oriented architecture except more expensive and complicated.by charcircuit
- There things are always a clusterfsck compared to the mainframe deployments.by kev009
- Ahahha so true man!
Some CICS regions, a DB2 and a couple of VSAMs and that's it.
by vb-8448 - yo, 2022 called and they want their buzzwords back:
https://news.ycombinator.com/item?id=32023863
https://wso2.com/engineering-platform/developer-platform/doc...
by amne - I wonder how they ensure durability. Is it possible that a cell going down would roll back a payment after it has occurred. Or do they depend on a non cell database?by neerajsi
- I would assume nothing related to a given transaction crosses the cell boundary.
We use a cellular architecture to help constrain the blast radius of a modular monolith. Each one of our customers lives in exactly 1 cell. Any kind of cross-customer BI/reporting happens through a data warehouse.
- American Express tech is some of the worst in the world among big companies. All of the value in the company is just in the branding. They put some work into the mobile app and the website, but other than that, its a facade.by llmslave
- What are you basing that statement on? It has not been by personal experience.
- BoA reauthorized an auto-payment card even with the card being expiration and uncorrected security code. I would call that authorized fraud by Bank of American.
This is why I find it best to declare a card stolen right before expiration or after.
by yndoendo - Having worked at Amex and other huge banks, let me assure you that there's much worse than Amex. Amex's Fraud analytics team was good. Risk was good. Ben's team is good.by jmpman
- A few years ago someone kept signing up for loads of bank accounts/credit cards in my name, with my address. I’m not sure what the point of it was. But while everyone else happily sent cards and stacks of welcome paperwork to me, Amex were the only one that contacted me and told me they’d detected something weird in the signup. They gave me some helpful advice to resolve that situation too.by mcintyre1994
- 403 Forbidden
Because of the title I was expecting to read about doing payments with a distributed network, like a terrorist cell network, or something like Hawala. Not (as I infer from other comments) Amex using multiple independent systems.
by inigyou - This matching up with sliced data, either sliced by recipient, or sliced by sender, so it makes sense to have primary state with cool backups. Particularly if the state is race-tolerant.
As long as event orderings are unimportant, or self resolvable this works really well.
e.g. if Events A,B arrive, but A+B => C and B+A => C, then as long as you durably record A, B, the end state is the same.
I'm not sure why "reroute" is a message instead of a response, I would expect it to be a failure response pruning the control flow. With the GTR being "default retry".
There's a lot of learned experience in that doc. Reading between the lines, both the logging and configuration systems have caused global outages (or near misses). Nifty to read.
This style of architecture fits with a "no global changes" and "never lose it all" approach to fault tolerance, accepting that there will be visits from "Mr. Cock-up" [1].
Very nice writeup.
by jpollock - microservices / clusters / zones - really all of these are other "cell-based" architectures as well. there is absolutely no written rule that a microservice was just an API or a singular service, it basically can be a independent instance that is testable/usable/gives value on itself.
that said: still a nice write up, learning about some of the architectural choices that AMEX makes is definitely insightful (and relavent/useful to what i am working on right now as well!)
by mkhalil - All i can see is a giant single point of failure called the Global Transaction Router.by nightshift1
- Isn't it reassuring the entire CC business hinges on a single proprietary appliance inside a cage at some DC? :)by manoDev
- GLBs aren’t SPOFs. They are typically deployed around the world redundantly, often using Anycast IPs or using DNS geographic and failover records, and are stateless. Think AWS Global Accelerator and Route 53 as an example. The architecture diagram is a high level simplification.by otterley
- Some of it sounds like it reinvented Erlang supervision trees https://learnyousomeerlang.com/supervisors. As a joke there we’re calling gen_severs “nanoservices”. Granted, that was mostly when microservices were the hot new thing.by rdtsc
- Immediately my first thought as well. I still keep coming back to Erlang/Elixir/OTP as the best possible choice all things considered. I should use it more.
- Whole lot of nothing.
This isn't about payment technologies, it's not about isolating transactions, it's about scaling the middle layer. What's worse it's not even explained what middle layer does.
No info on how routing works, no info on data synchronization.
Folks just learning Kubernetes and write extremely abstract stuff.
by Insimwytim - I agree, confusing and it seems like they are using the coined word "cell" to describe "container", but they really should say that instead of making up new words.by jijji
- The problem is there is no incentive to talk about things in simple terms, in both companies and academia. If you make your work look simpler, it is assumed it is easier and thus your contribution is less. There are incentives therefore to make your work look more complex than it is.by zipy124
- Nobody uses Amex for payments, so the system isn't ever under high load.
Just kidding!
I find the idea quite good, and have to assume that the amount of payment fails they experience due to partitions/outages isn't very high and that the post-payment reconciliation and reclamation process gives them the liberty to rank availability a bit higher than correctness.
One thing that looked a bit shaky was the interplay between the global transaction router's state of knowing which cells can handle a particular payment and the asynchronous distribution of the "failover data", which I presume it needs to know to route correctly. To me that seems to create a window where it might route to the wrong cell due to an outdated routing state.
It also doesn't go into the HA setup of the global transaction router itself.
But still, I kind of like the design.
by physix - Amex is gaining popularity for acceptanceby whazor
- >To me that seems to create a window where it might route to the wrong cell due to an outdated routing state.
But if the router sends to the wrong cell the cell will either send it back to be rerouted or it will fail and the router will try again (or report back the failure so upstream can try again I assume)
by mixdup