Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- This is kind of beside the point but... Shouldn't the the evolutionary metaphor be the opposite of how the author's using it?
They go on about how we've settled in the trough, but the whole local optimum problem is about settling in an early peak, and being unwilling to cross a trough to get to a higher peak. The workarounds would be propping up the first peak, not evidence that we've settled in the trough.
Either way, the metaphor feels kinda forced IMO.
by ksymph - You don't really need a new protocol. If you trust your consumers, you can just give them a paginated "/events" endpoint and have them poll that. The delay will be variable, but tunable, based on the required timeliness. If you want more prompt responses you do long-polling or a websocket.
The key, and only thing that matters, is that the cursor rides in your database, and is therefore transactionally consistent with the event. That's the whole magic trick.
We've done event streams like this at the bank I work at for years.
by delusional - Just treat webhooks as a hint.
Write your code to work as a "reconciler" that checks the state of the remote system and reconciles it with the local view of that system. Run reconciliation for the full state periodically using a scheduler, and then treat webhooks as a hint to run the reconciler immediately.
This way, you will have a robust system that can survive logical bugs and outages because you don't store the synchronization state per se.
In the case of Stripe, for example, have a process that polls every open checkout session every couple of minutes. A webhook then just triggers the run earlier. If you're worried about DDoS, have an exponential backoff for the poll period.
Theoretically polling doesn't scale, but in practice it works just fine.
by cyberax - The core tradeoff here is "Push vs. Pull". Webhooks are a model for pushing data to subscribers. A traditional API allows clients to request and pull data. The data flow is in the same direction, but control flow is opposite.
Pull-oriented models are much easier to reason about and should be preferred where possible (cybernetically they are a closed loop, vs. push models which could literally just be a barrage of UDP packets). But they do have a little bit of overhead which makes them the wrong tool for some cases, like live-streamed entertainment or massive telemetry flows which value performance (latency, throughput) over missing a few packets.
by pphysch - You mean pull-oriented models are easier to reason about, right?by lobofta
- Webhooks are a painful problem. To clarify, Stripe's events API definitely ships a cursor and polling it has been the method preferred by large consumers for a long time.by zrail
- Stripe events API is one of the examples of how to do things properly. And SCROLL is just trying to create a common spec so that everyone offers a stripe-like event polling api.by weli
- With webhooks, consumers get to asynchronously respond to updates from a provider. If no data has changed, a provider will not send any updates.
With SCROLL, consumers are responsible for choosing when to ask a provider for updates. Without a mechanism for knowing when data has changed, consumers will be forced to be pessimistic and poll providers for new data on some cadence.
I see two issues with the proposal: (1) SCROLL will lead to an increase in unnecessary network traffic for both the consumer and provider, and (2) because a consumer cannot know when data has changed, the lag between a consumer's local model and the provider's data model will be larger when with Webhooks.
by zffr - If your API is just wrapping Kafka, it can long-pollby inigyou
- Assuming you're not using the proposed streaming option, I suppose you could always send a webhook for that fact alone? In other words, an empty notification, with semantics of "something has probably changed, better poll the SCROLL if you aren't already".by lxgr
- Webhooks are simple and ubiquitous, and that's both a weakness and a strength. It's also why they are used for a lot of things, even things they are not great for (state sync).
These weaknesses are why we[1] added FIFO endpoints, Polling Endpoints, and what we call "Svix Stream" as ways to do ordered state synchronization (each with its own tradeoffs). This lets people consume the events in the way that best fits their use-case. We are working on more things to make the state sync even easier. I'd love to hear about more challenges people are facing with webhooks, as we want to make these things better.
OP: I'd love to hear more about your thoughts there, and will send you an email in a moment.
P.S, if you're unfamiliar, please check out Standard Webhooks[2]. It's a spec we created to help with signature verification that has been adopted by OpenAI, Anthropic, Google, and many others. We are chipping at one webhook challenge at a time. :)
1: I'm the founder of Svix (mentioned in the post), we do webhooks infrastructure as a service.
by tasn - Gerard mentions it super quickly, but another massive issue with webhooks generally is local development. Yes, you can use a tunnel, but that requires all engineers on a team to add their own tunnel urls. This causes even more issues when you use the platform as a source of truth, like for auth or payments. With WorkOS specifically, your whole team develops with one shared development sandbox. You run into issues when your local dev auth (in postgres) is not synced with the shared dev sandbox that WorkOS has since not all team members have their dev environments running at once. So yeah, then you use events API. But WorkOS only preserves the events API data for 90 days (and u have make 3 calls since its a max of 30 days per call). So then you load all the data with the state API first, then you start running the events API. It's a mess.
Tried to talk about this on X until the CEO of WorkOS wanted to bring it in private, then proceeded not to help at all. https://x.com/grinich/status/1913035839866835297?s=20
by shreygupta - With the proposed solution every consumer will have a persistent connection to the server irrespective of the frequency of events. This setup seems inefficient unless you have a very high volume of events coming in. Many CDN networks have a limit on how long a connection you can open. And data providers will not prefer serving persistent requests.
Problems listed are signatures, dedup, buffering, bootstrap, cron. Everything other than signatures and bootstrap, can be solved by having a counter in every webhook payload. It will increment each time. When you receive a webhook and the counter does not match, the consumer can fetch the missing data from the events API.
I agree with the author that providers simply saying "at least once delivery" is insufficient. they should have solutions that does not require an architecture diagram.
Bootstrap is better served with a bulk events API so you don't make one call per request. It can have an after/cursor pagination. Solutions that work for our internal Kafka might not be suited to work across services, over the internet.
by bytesandbots - Open TCP connections can also be wildly cheap and efficient - Apple Push Notifications (APNS) and Android's push systems maintain open TCP connections to just about every mobile device on this planet.
An open connection is just a bit of state on either end. The C10K problem has been solved for ages.
Anyone remember consuming Twitter hoses back in the day? Those were also long-lived persistent connections for efficiency reasons.
by oasisbob - I much prefer cursor paginated API requests vs. webhooks. The obvious downside being that in order to not get 429'd you need a respectable poll frequency - meaning you lose reactivity to new events.
Thus I think webhooks still have a place - but as a simple "poke" that can be sent to the client to tell them something has changed - supplementing a default low frequency polling interval.
This gives us the best of both worlds:
1. No need to bother de-duping/retrying pokes - if you miss a webhook you will shortly recover anyway when you next poll. 2. No need for any local-specific tunnelling/tooling - the local app will work just fine with the default poll interval. 3. No need to keep a connection live for each client. 4. All the good stuff OP mentioned in his blog post.
by tlonny - Absolutely. I have been so livid at so many applications for not providing a decent CDC API (and dont forget deletes). Salesforce perhaps is the best out there. Imagine if every application exposed a standard CDC API, the world of integrations would be so much better.
Webhooks are fine, but a pollable API is a must have. The amount of hacks I had to do at work to workaround shitty APIs gives me nightmares.
- Yep, the poke pattern is additionally nice because it means my state reconciliation function is the same when running on cron interval vs event driven.
On the flip side, it helps to have endpoints which have a query param linking to some sort of resource update time stamp. That way you can query to only get those items changed since last poll.
by evolve-maz - The Gmail API works nicely like this. There's a history.list endpoint where you can see the recent history of message additions and removals, you can query for just the history that's taken place since a specific event's historyId, and you can subscribe to push notifications (that can be delivered by webhook) which just tell you when there are new history events, and you're expected to hit the history.list endpoint to see what's new. Some dropped push notifications aren't a big deal.by AgentME
- I had the exact same thing with the Quickbooks api recently. You cannot trust the responses or webhooks at all.
On create a user or invoice for example sometimes it will return an error, yet it actually created the entity. This means you have to check manually after creating everything to know if its created properly.
Then you have the issue that sometimes quickbooks takes a while to update, and locks the company file while it does some background magic. This means you cannot immediately do the existence check, and also sometimes the check errors or times out which essentially means you need to keep checking forever until you can properly reconcile your db against theirs. But with hundreds/thousands of transactions per minute this state is never reached. You perpetually live in a state of trying to catch up but never managing it.
When I brought it up with Quickbooks dev support their response was literally "Its your job to make sure things are created properly in our system".
How did we get to this place where we started putting up with systems that cannot ever be trusted?
by alt227 - Easy, distributed systems, coupled with folks that think they are the solution to what should be properly written modular applications in first place.
Instead they deep dive into distributed systems, without ever learning about them and all the issues that can arise.
There is a reason many CS degrees have two semesters full of distributed systems content, between networking approaches, architecture design and algorithms to make everything robust.
by pjmlp - > You cannot trust the responses or webhooks at all.
Well... yeah. I mean it's pretty obvious, no?
Here's some things that could go wrong regardless of what care the software tries to provide:
- The transaction completed on the backend cluster but the app instance died before if could create the response and after it committed the transaction.
- The transaction completed, the app instance transmitted a response, but the load-balancer/reverse-proxy in-between died before it could relay that response.
- Everything went well, but the ISP dropped some packets before it could get to you.
- Everything completed and the ISP stayed up, but on your end the response was flagged as malicious, or never made it through your load-balancer.
So, yeah. in general when you make an API request and get an error you have to check if the state was changed anyway, and if you aren't doing that you're doing it wrong anyway and cannot blame the system on the other side for returning errors.
by lelanthran - That is the way enterprise software works as a system. It demands to be the central focus of everything. Workers want to route around these turbo productivity theater nonsense that could be replaced by a few K script that gates access to a text file and checks validity of appends. That can’t be allowed, so you need what is essentially whole poorly documented OSs to enable an economy of brokers to it, or the whole con would collapse.by hyperhello
- This is a nice writeup of the problems in using Webhooks for State Synchronization. I also noticed that the proposed solution is a pseudo IETF-style draft protocol called SCROLL... that happens to be remarkably similar to an actual IETF draft I am bringing to IETF 127 this November called "Braid-HTTP Subscriptions."
Both drafts request a subscription with a GET plus a header:
In both systems, the GET leaves its response open to stream events. SCROLL responds with application/x-ndjson. Braid subscriptions are a 209 Multiresponse, with content-type application/http-history. This lets them support more than just JSON. You can send updates to the state of CSV, or PNGs, XML, HTML, plain text, or any media type.Scroll Request: GET /scroll/feed/customers Prefer: stream Braid Request: GET /customers Subscribe:The author noted that it's hard to get adoption. Well, the reason that Webhooks are so common is that they are bog-standard HTTP. For this to get adopted, we need to put it into bog-standard HTTP. So we need to go to the IETF, and and extend HTTP in a general way to support state synchronization. It should just work for any existing HTTP media type (not just JSON), and any resource/URL (not just special /scroll/* URLs), and any way of marking timestamps (not just the ordered strings proposed in SCROLL).
Then we can bake this stuff into HTTP, and thus into all our bog-standard libraries, utilities, and code, and you won't have to reimplement the same sync-logic-over-webhooks again, and again, and again.
Reach out if you're interested!
by toomim - I'm sorry, but I have to ask. bog-standard?by angrysponge
- the solution is good but what tends to happen is webhook providers are not gonna work on such a solution - why - because it puts 'work' on them.
hell this is without the proposal for a new protocol - just a 'GET' stream or paginated one like the author said.
whereas with web hooks - a consumer has to do all the work - as the article above outlined.
by dzonga - I reached out through email :)
Just one correction. My spec doesn't force /scroll/ URL's, just proposes it as a convention.
by weli - Is it accurate to say this is something like long polling except you continue to hold the connection open for subsequent updates? Does this mean a server potentially needs to hold open a very large number of connections (one per client) even if there are no updates?
And why formalize on HTTP rather than on a similar protocol over websockets?
by bobbiechen - There’s also the Linked Data Event Streams (LDES) standard, which is a way of hosting a log as a set of linked http documents, fetched via polling and link navigation. Is there really anything more needed than a webhook for the notification and an LDES for the log?
https://semiceu.github.io/LinkedDataEventStreams/releases/1....
by Joeri