Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- it looks great lm gone try this outby Brikuio
- There are two pieces. An SDK that runs inside your service, and an MCP server your coding agent talks to. The SDK is what makes setting probes (virtual breakpoints, log or metric) possible without a redeploy. In Node and Python it hooks in-process. In Java it attaches as a JVM agent, instrumenting at the bytecode level. Either way the service keeps running and serving traffic. Nothing pauses.
Consider putting this near the beginning rather than 2/3 of the way down your pitch. I nearly stopped reading because these dramatic 1-2 sentence paragraphs are unpleasantly like listening to TV commercials. I think your target audience should not be CTOs or their direct reports, but engineers themselves, and I think you need a more focused pitch that takes less time to get to the point.
Anyway, an MCP-managed passive debugger seems like a useful tool. Best of luck with it.
by anigbrowl - Appreciate the feedback, will try to highlight the how instead of why specially to devs
Although, our primary sell is debugging, the context from production on how things work currently helps ai agents during feature development and code reviews as well.
by karanraina - Congrats on the launch! The ability to drop read-only probes into a live service without triggering a painful redeploy is a massive time-saver. Since my workflow relies heavily on cloud-based development, I am curious—how does your SDK handle serverless environments where the container lifecycle is extremely short? Really great concept!by MdJasimuddin
- Thanks!
You're correct, serverless is a bit tricky. CPU gets suspended the moment your function returns. The way it works is that you wrap your functions with a wrapper in our sdk.
that wrapper is supposed to track if there's telemetry to be sent, if so.. it sends it, otherwise, return as usual
this makes sure that when there's no active probe, there's no latency added. But when there's an active probe.. ~100-200ms could be added in the worst case if the probe is just before the return.
again, this isnt a problem in non serverless worloads because the CPU is always on.
but since probes are bounded by time and count, this will go away as soon as the time or count condition meets. beats adding new logs and redeploying in my opinion
by karanraina - Looks cool, do you have plans to make an open source version for on-premises installation?by denis-stable
- we currently support on-prem setup ourselves. You can self-host Hyperprobe with redaction so all the data we capture never leaves your environmentsby shailendraht
- Spot on and nice complement to the pre-merge half of this problem. Silent logic bugs slips through when a diff "looks right" — what you're building catches them once they're live.
- Congratulations on the launch. Positioning this as an AI-driven debugging layer on top of existing observability tools makes sense, and the read only probes for silent failures feel like a practical way to get runtime evidence without turning every incident into another log and redeploy cycle. Will give this a try for sure!
- thanks for the confidence!by shailendraht
- Congratulations on the launch. I love the idea and the execution.
When I looked into this a while back I explored using ptrace() to add breakpoints and even add functions at specific line numbers. But ptrace is so slow, and it doesn't work with bytecode-in-VM setups.
What were some of the requirements you guys had when building HyperProbe? I can see low latency was one.
by kirtivr - ptrace is too low level and will freeze your process. it cant be used for production debugging i feel.
we work at the application layer by hooking into production grade tooling if available (inspector in v8, sys.monitoring in python) or bytecode manipulation(jvm)
since we arent controlling the application for a different process, we dont need to freeze the app to get the current app state
requirements we had in mind in order of importance
1. safety -> user app needs to function as usual no matter what happens, there shouldnt be an error in the user's app because of us
2. zero idle footprint -> if no probe is active, cpu/memory differency in the user app should be immeasurable
3. zero latency footprint at non probe paths while other probes are active
4. measure mem/cpu footprint directly or via a proxy like eventloop lag and have guardrails around it. suspend probes or even lose snapshot data if guardrail conditions meet
5. minimal mem/cpu footprint for active probes
6. minimal latency foot print for active probe paths
by karanraina - This debugging in production thing has always been interesting to me. Rookout, etc.
How does it work? Using the NodeJs inspector API or other language equivalent to drop breakpoints? Those APIs are unavailable in many serverless environments and are challenging to use alongside bundlers.
- YES!!! for nodejs, inpector API is used. But if you're adding dynamic logs or metrics, we dont even call the inspector completely. we return an expression that will always evaluate to false and safely evaluate our the log/metric. saves time and computations happen in the same cpu cycle
You're correct inpector API is not available in many non-v8 targets. Bun also has somewhat of a partial support for inpector API but at least has a programmable debugger interface. It's not going to be as fast as native inspector but its better than nothing i guess :P
for python sys.monitoring. for JVM, we do bytecode manipulation itself.
bundlers are not an issue because we support sourcemaps. We just need mappings, not code in the sourcemaps and we do sourcemap resolutions out of process so that your app doesnt spend ~200 MB of memory for parsing sourcemaps
by karanraina - Congratulations on the launch! Looks very neat.
For people that don't have these neat observability tools (like me), I've been using https://shellshare.net (disclaimer: I made it).
This is a single command to share a terminal live with e2e encryption. Originally it was for teaching classes or helping colleagues, but it's also very helpful for agents. I SSH into prod and run:
> npx shellshare exec --json -- tail /var/log/my-app.log
This generates a URL, then I can tell any agent:
> monitor <URL>, instructions in https://shellshare.net/llms.txt
They can see the output live. No need to install anything in the agent's machine. Next shellshare version it will be just "monitor <URL>" and the agent's instructions will be in the URL itself.
Nothing even near what you've guys done, but it has been helpful for me. Best of luck in your startup!
- Congrats on the launch. Two things I'd want to understand before putting this anywhere near a hot path:
1. What makes "read-only" a guarantee rather than a convention? In Python a plain attribute read can hit a @property that lazy-loads from the DB; in Java a getter can mutate state or take a lock. If the capture expression permits attribute access at all, read-only becomes a property of the code being probed rather than of your SDK. Do you restrict the expression grammar, or is it best-effort?
2. What's the shape of what comes back through MCP? A captured frame can serialize into something enormous, and an agent will cheerfully spend its entire context on one request object. Can you project at capture time (user.id rather than user), or does trimming happen after the full payload is already built?
by akashy123 - How is HyperProbe different from existing tools like AppSignal, Rollbar, and Embrace? Such very mature tools exist that auto-instrument, collect variables from the call stack, and pinpoint error causes.
> Every log-and-trace tool hands the agent data that already exists and asks it to reason backward to what probably happened
If the app is using a decent instrumentation tool, the data shows what 'actually' happened, not what 'probably' happened.
> "checkout returns 200 but some users are seeing their order fail, find out why."
Does this tool only exist to shore up poor system design? Failing orders at any e-commerce business I've worked with, large and small, are a huge red flag. Typically that is one of the first actions that is logged and traced (alongside onboarding/login), and the metrics are actively monitored. Returning 200 for failure and not catching that error is very bad API design.
Similarly, putting engineers in a situation where debugging requires accessing unknown amounts of live sensitive customer data is generally considered bad practice (even if it happens often IRL) -- in a hurry to debug, it's easy to miss that a property should have been redacted; by then it's too late and sensitive data is exposed. Plus, in most systems with significant usage the volume of trace data is prohibitive to individually examine and search through. That's why Rollbar etc aggregate errors and captured data to identify patterns before a human (or agent, or tool) ever takes a look at it. A single captured instance can also be very misleading as to the true cause.
How are you addressing these common concerns?
by doublerebel - > How is HyperProbe different from existing tools like AppSignal, Rollbar, and Embrace?
These work only on either uncaught exceptions or wrapping up caught exceptions with their sdk. These tools will not help you with silent failures, like logic bugs where code executes cleanly without throwing, but produces the wrong business state. If every problem in your app ends up as an exception, sure you'll be able to catch the symptoms of where the exception got thrown. we can deal with these too, but these tools cant deal with the messy bugs where no exception fires.
> Such very mature tools exist that auto-instrument, collect variables from the call stack, and pinpoint error causes.
That is true for python using frame.f_locals (we use this as well)
nodejs only gives it only till the lasy async boundary, after that v8 itself drops this data. java only gives you the current frame, to get variables beyond that you would needs JDI/JVMTI which would block your threads, usually unnacceptable in production
To get around this safely, we add multiple probes all across the call chain and collate collected data using the traceId from the context (or thread id as a fallback);
> Does this tool only exist to shore up poor system design?
Returning 200 OK on a silent failure is 100% bad system design, I completely agree. But real-world production systems are full of legacy edge cases. (if that weren't true, L1/L2/L3 support team shenanigans wouldn't exist)
Also, the exception will tell you that an exception occured in order service in GET /orders/{id}/payment, your trace will tell you payment service is giving 404 for that order ID
what it wont tell you it happened becuase the webhook endpoint that your payment gateway calls is now receiving a new payment state called 'PENDING' and that you dont handle but still mark the payment as 'processed' for idempotency check. and now your order service is calling the payment service and its giving 404 because it never got written
Bad design. 100% Agree, but has happened IRL.
> putting engineers in a situation where debugging requires accessing unknown amounts of live sensitive customer data is generally considered bad practice (even if it happens often IRL)
I think tells that teams would go to these extents to fix issues. Not ideal. I agree.
> in a hurry to debug, it's easy to miss that a property should have been redacted; by then it's too late and sensitive data is exposed
fair critique. we currently use in-process rule engines to filter known sensitive patterns, and users can add on to it. but we are also building out-of-process secondary checks (using NER/classifiers) to sanitize payloads before storage. It requires strict rules, but getting verified runtime evidence is far safer and faster than blindly guessing and shipping trial-and-error hotfixes to production. or waiting to be too sure.. a luxury that might not be possible everytime.
> Rollbar etc aggregate errors and captured data to identify patterns before a human (or agent, or tool) ever takes a look at it.
There is merit in that as well, if you are looking at so many logs/traces, you kinda have to do it. We have a different approach, we use hypothesis driven conditional probing instead. probes are dropped dynamically as the understanding of the bug evolves in a session
exmaple:
console.log('hello');
const x = await getThisValueSomehow();
if (condition A) {
console.log('i m in condition A');
// do something;
} else if (condtion B) {
console.log('i m in condition B');
// do something;
}
You can also place a probe before the branch to capture variable state when neither condition evaluates to true. You gather precise data on demand rather than paying to store petabytes of static trace data.
> A single captured instance can also be very misleading as to the true cause.
We collect multiple snapshots per probe run. However, because we capture full variable state at the exact execution line, a single snapshot frequently reveals the root cause for that specific failure path. If that snapshot raises new questions, you/your agent simply drops more probes deeper down the call chain
Thanks! This was very insightful
by karanraina - Two things I would want to know before pointing this at a hot service: (1) the overhead budget — when a probe lands on a hot path, is capture sampled or capped per hit, and what p99 latency delta have you measured under load? (2) failure isolation — if probe evaluation itself throws (weird object shape, getter with side effects, huge captured value to serialize), is it contained so it cannot take down the request it is observing? In-process agents live or die by staying boring under worst-case conditions.by tizerluo
- we have a lot of guardrails (https://docs.hyperprobe.co/how-it-works#built-in-safety-guar...)
if any guardrails fails, we suspend probes till cooldown.
also 1. every probe is bounded by hits/expiry time (whichever comes earlier) 2. hit budgeting happens with a token bucket at a global level, per probe was an overkill (numbers are configurable) 3. we even measure the execution time that probes have when active and suspend if that that takes longer than threshold (again configurable) 4. we even have budgets for the network bandwidth it would take (approximated by the size of payloads) 5. collection itself is bounded by max no of total snapshots we can keep in memory. 6. every snapshot has a size limit as well, every variable has a size limit as well. 7. depth of objects, no of objects, size of lists is capped by default.
latency delta varies by platform under load but is mostly negligible
nodejs: ~7-10ms python: ~4-9ms java: 1-2ms
the main reason for this is guardrails suspending probes, having loosened guardrails will increase this under load
regarding localization of failures.. absolutely we even report the error in the probe snapshot (confirmed by adding side effects in an expression and commenting out the guardrails during testing)
huge payload size doesnt matter.. we limit the objects depth, list length, remove duplicate refs from data etc.. even string length is truncated., but even if it happens, your request would still survive.
also, even if the collector dies or there's a network failure, your service remains unaffected, we just are unable to collect telemetry
we are boring under extreme conditions :)
by karanraina