Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- I looked at the OTel schema generated for recording a single numerical metric. It was like 12 or 13 meta fields in addition to the actual metric fields like timestamp, metric and value.
OTel looks like something designed by a committee of committees, funded by someone who is in the business of selling cloud storage/data warehousing services.
by psadri - It really never grokked with me why there isn't just "open source Datadog" that can be installed and used. End to end, stateful, that we can just self host.
Our team tried to set up open telemetry to replace Datadog and got totally crushed in complexity. The model of having Open Telemetry just be for standardizing & exporting to other backends, needing glue for each part of the setup was nuts.
by czhu12 - Signozby luckydata
- clickstack https://clickhouse.com/clickstack
is the closest i've seen to the datadog experience
by arccy - Signoz?
But yes it seemed like OTel was more interested in being a spec than a tool.
by reactordev - isn't this exactly what the LGTM stack is?by frez1
- I run OSS Grafana with Loki, Prometheus, and Tempo. I use an Alloy sidecar taking in OTEL and scraping logs.feom my Go services and selfhost the stack. Once you need to scale it gets a bit more complicated but it's all still OSS.
The biggest challenge I have is that each data source needs it's own query language, which DD and the like don't. That's why at my day job they went with DD despite the costs. Still OTEL but the querying is the same. We are also looking at Dash0 but for all of my personal and consulting jobs, OSS LGTM/P works good for me.
- There is, it's called VictoriaMetrics/Logs/Traces.by pphysch
- I find the entire observability space to quite a poor experience, at least in the self-hosted space. Tried both grafana route and signoz and neither seems particularly pleasantby Havoc
- Try datalust/seqby N_Lens
- IMO Grafana is definitely the best o11y tool when you are just starting out. But it isn't built for scale. When you have to analyze petabytes of logs, something custom built for this like CtrlB might be more helpful.by csujoy
- What about the experience did you find lacking?by nunez
- OpenTelemtry is the perfect example of an overengineered mess.
While I usually think that at least having some standard that people agree on I think OpenTelemtry should be dropped.
A lot of the less popular alternatives (just going with Prometheus, Victoriametrics, etc) are de-facto competing smaller standards and a lot better both in terms of less added complexity and the results you get.
I think OpenTelemetry turned metrics into a farce. In many situations even self-rolled telemetry works better even with the added stuff. The annoying thing is that OpenTelemtry is that big standard now one kind of has to to add compatibility. So please, if you write software, make sure you don't lock yourself into OTel.
by tete - I agree overall, however:
> A lot of the less popular alternatives (just going with Prometheus, Victoriametrics, etc) are de-facto competing smaller standards
By all metrics (hah), Prometheus is the more popular solution and is the de-facto standard, as far as I know.
by nlitened - The alternative is vendor lockin, $$$, and spotty support for complex environments with zero chance of ever getting 100% coverage.
At least with Open Telemetry, anyone can write an OTLP "source" using free, open specifications, and it'll "just work" with dozens of third-party "sinks". That's huge!
Sure, there's a lot of experimental tags on semantic conventions, but at the end of the day, that's not that critical. It's just data: most sinks don't "interpret" these tags, they just display them as-is, so changes aren't breaking changes.
by jiggawatts - The alternative is Prometheus (which is freaking great) and Jaegar (which is freaking great), each alone. This is better, because Otel is trying to put two distinct things (monitoring and metrics, distributed tracing) into one package, because they know how to use neither.
Neither Prometheus metrics nor Jaeger traces are magic bullets. Neither of them are complicated, either, and in fact the fact that they're not complicated is their greatest strength. You can and should understand every facet of what they entail. You should build the (very small) shims that they need for your company's framework every time. It's not hard. It's not hard because it's not complicated. The fact that it's not complicated seems to break people's brains. They are accurate because they're simple and they're easy to work with because they're simple, and OTel is neither.
- Its a shame that the various implementations are pretty horrible. Global state, static methods etc etc.
If you get rid of that, and just pass dependencies around, create some appropriate local abstraction around them.. the tooling, be it datadog or honeycomb does a great job making it useful. Can't really say the same for grafana, but ymmv - depending on budget
by time4tea - Sounds a lot like K8s. It's not a framework you use, it's a framework to build a framework on top of.
I wish the observability vendors would move to using it under the covers so it's easier to mix and match.
I wish the otel support wasn't super buggy in most of the frameworks and backends.
by rcleveng - But then you wouldn’t be locked in!
- OTel is so frustrating. If it wasn't shaping to be the clear winner in the space, I wouldn't complain about it as much. But today:
1. Every major vendor is still in some weird alpha/beta support for OTel even after all this time.
2. The performance hit is substantial and makes you question what the point of performance instrumentation is if you need twice as much compute/RAM to run the same workload now.
3. Serverless runtimes pay a heavy penalty for cold starts with OTel.
4. You're basically forced to run both gateway collectors and edge collectors for any realistic usage.
5. You still need to configure destination exporters in unique ways. This leaves you questioning what the value of OTel was.
6. Vendors that go beyond the scope of what OTel covers still need their own bespoke instrumentation. What was the point of any of this then?
by bilalq - So what's the alternative then? (Genuine question, not hypothetical snark.)
- > 4. You're basically forced to run both gateway collectors and edge collectors for any realistic usage.
You most certainly don't. You can run your app (especially if it's "serverless") without the collector agent.
App-to-agent and agent-to-sink use the same protocol, so all you need to do is set up the tracing/logging/metrics exporters to directly speak with the sink. These days, it typically means specifying the URL and the DSN header.
by cyberax - I've never found instrumentation to be a huge issue. Sure it takes more effort but you get a lot more value once you understand _business_ events.by brikym
- I disagree. I'm an observability geek, and OTel is... fine.
It's missing a few things that I'd like, but I was able to implement them myself. I guess the major design issue is that the sampling decision is made at the _start_ of the segment. So I hacked up a few improvements:
1. Ability to mark segments as "boring", so they are dropped before the export. For things like healthchecks, empty "get the pending jobs" queries, etc.
2. Ability to downgrade errors for segments that are expected to return an error (e.g. HEAD on a non-existing object in S3 to check if there's a cached blob).
by cyberax - We did the same for 1, we have a service that picks up messages from a queue and drops 95% of them. In Python, at least, it was really really ugly. We had the alternative of using refinery, since we're on honeycomb, but didn't want to get stuck with something that only works with one vendor.
I don't understand 2, though, that should be easy? You have access to the span and you can set its status code to 0. With the Python SDK that was trivial for us to do.
The one thing I really hate about the Python SDK is that there's no simple way to say "attach this attribute to the top level span in this app". That way, when you get the user id further down in your app, you can just annotate the top level span with it. Thankfully, honeycomb has any, any2, root, parent (that allows you to filter on any span in the trace, or on the root or parent) but that's slow and again, vendor specific.
by gcbirzan - Agreed. Otel itself is fine. The documentation is bad though and full of inconsistent best practices and examples that are flat out wrong and other things.
My life of working with it got easier when I started just looking at the actual code, using network level tools like nc/tcpdump, making extensive use of the debug exporter, and almost ignoring the docs entirely except as a basic summary of what a thing does.
by chrismarlow9 - OTel has a nice property of working, which sometimes can't be said about vendor stuff.
I think the actual APIs kinda smell at the language level, and when Honeycomb decided to lean into otel and deprecate its Python libs I was super sad, cuz HC's libs were _way_ more usable IMO. Docs are also... painful. Real painful.
I wish that I could get a Python lib which is like "here this is Otel but the config phase isn't weird, and the API just looks a bit better". One of these days.
The biggest trouble I have with Otel recently is getting fixes patched upstream in contrib. Using contribs is super dangeerous, and I would basically recommend people write their own instrumentation and treat the contrib packages as just examples of how to do it
by rtpg - HN always grumbles about OTel, but I agree. It's fine, and important: https://jeremymorrell.dev/blog/opentelemetry-and-the-value-o...
I understand the author's perspective in the linked article, but none of that data shows a project in trouble? Some languages have more resources than others, but those all look like healthy open source projects
by masterj - What always puzzles me about OpenTelemetry is that tracing, metrics and logs are all designed independently. I wish there was a way I could just annotate my code base once, and let the ultimate decision to expose something as a metric/log/trace be dynamic at runtime.
For example, if I look at a graph in monitoring dashboard and see something suspicious, I’d like to say: “The next time something like this occurs again, please save me a trace.” I should be able to just do that with a single mouse click.
I remember them releasing the tracing spec/SDKs and saying “now let’s move on to metrics/logs.” That never sat right with me.
by EdSchouten - One strategy do to do that is to trace everything by default and select what to sample later, e.g. https://grafana.com/docs/grafana-cloud/observe-and-act/adapt...