Martin Thwaites

Quick reminder that people are still finding the fediverse, so keep coming back to you contact lists on the various sites you follow them to see their new handles. Mine is @martindotnet@hachyderm.io

@davidfowl My system prompt came from the OSS community, your system prompt is biased towards Microsoft.

So when programming languages are all irrelevant because we’re slopping out code in any language, what are we going to replace “my language is better than yours” discussion? Will it be agent harnesses?

@bencera @GeoffreyHuntley @honeycombio Thats why we built the MCP, now you don't have to open the site to lose that time, you can lose it right from your IDE or console. Now THAT'S progress.

@GeoffreyHuntley @honeycombio honeycomb is one of those tools where you open it to check one thing and 2 hours disappear

one reason why i love @honeycombio <oh that data looks interesting - click>

@adamjonwilliams @GeoffreyHuntley @honeycombio Its coming :)

@GeoffreyHuntley @honeycombio they should just replace their whole site with this and a pricing page, would at least be more clearer what it does

@cloudsider @Aaronontheweb @ChetHusk The reason comes down to "when" Aspire (the orchestration part) injects the environment variables in general (not just the endpoint), and whether it does that before or after the Collector injects it's environment variables via an Event Hook.

@Aaronontheweb @ChetHusk @MartinDotNet Is that because the dashboard needs to ingest the data and when using that env variable with otherwise default otel .NET settings it will only export to the sink endpoint from the env variable(?) Does it also overwrite the protocol to grpc if you set http?

Made a very short video (<3m) on trying to get the OTEL Collector to work with .NET Aspire, because I wasted several hours trying to make this work this week.

@Aaronontheweb @davidfowl @ChetHusk That's probably more of an issue with aspire in general so I'll leave that to david and the team. What I would say is that the method that was using is deprecated now, so that's probably a reason it shouldn't be fixed.

@MartinDotNet @davidfowl @ChetHusk It's also this though - I expect my environment variable values in the Aspire dashboard to be the real values

@davidfowl @Aaronontheweb @ChetHusk And in fairness to Aaron, he did actually file an issue.

@davidfowl @Aaronontheweb @ChetHusk I've just tested the one from CommunityToolkit and its working fine. I'm assuming it was a problem in switches we made to the eventing stuff.

@Aaronontheweb @ChetHusk I'd be interested if you could use the one from the CommunityToolkit, that one is a bit out of date (a few days) since we merged the community toolkit one. You could have also just pinged me to fix it :)

@ChetHusk @MartinDotNet Is there some sort of new annotation one can add to tell .NET Aspire not to hijack the resources' OTEL_EXPORTER_OTLP_ENDPOINT value?

@RogerAlsing @dddanielreis @dotnet @davidfowl Maybe you're in that niche, in regards to the question that triggered this, I would still monitor the raw inbound and outbound event data, over generating a pre-aggregation in the app. Increase the circular buffer limit if thats a concern.

@RogerAlsing @dddanielreis @dotnet @davidfowl For the specific "sent and received" like the original question was, I would suggest that does apply. Metrics tend to be useful in very niche scenarios, when you know what you want to query and how you want to visualise it.

#dotnet #testing @dotnet in .NET, using counter metrics within integration tests, is that a thing? Let´s say I have a metric for Sent Requests and another for Received Requests. would it be unreliable to subscribe to those in a test and ensure they are equal?

@RogerAlsing @dddanielreis @dotnet @davidfowl 10k Request per second is a non-rrivial amount, even in the IoT space. For gaming, especially in the game loop, tracing is likely not the answer. Applying this to the general case is the real problem. Most people are not gaming, IoT or hitting 10k request per second.

@RogerAlsing @dddanielreis @dotnet @davidfowl If you're at the scale of generating 5 useful spans per request, they're not easily replaced by rollup spans, etc. Then sure, maybe start thinking about head sampling first, and if that doesn't help, maybe think about pre-aggregation inside the application.

@RogerAlsing @dddanielreis @dotnet @davidfowl I've just checked and its been refactored into a circular buffer now with a default of 50k capacity. So yeah, if you're generating more than 50k spans per second in a single app, maybe think about head sampling, maybe, at that point, you might need some pre-aggregation.

@MartinDotNet @dddanielreis @dotnet @davidfowl but if the memory limit is reached, what happens on a low level? even if fully async, any overflow would then propagate to fill the thread pool / task queue instead. either the system halts, or it grows unbounded, or it starts to drop data, there are no other alternatives

@RogerAlsing @dddanielreis @dotnet @davidfowl From my testing (arguably a few years back know and a lot has been changed/refactored), console writing has back pressure due to the writing to stdout, with Activity is simply adding to the list and another thread pulling from it, so its slightly different.

@MartinDotNet @dddanielreis @dotnet @davidfowl Isn´t it very much the same issue here? any worker thread trying to write data to the buffer, buffer is full, thread is hard blocked (?)

@RogerAlsing @dddanielreis @dotnet Its not a fixed buffer, so its only limited by memory (similar to the way that console writing is buffered). In the event of a backend failure, that buffer will have an increased size, which is the same for Meters. The idea is that you offload it to the collector not a backend

@MartinDotNet @dddanielreis @dotnet so how does it work, what happens when the buffer is full?

@RogerAlsing @dddanielreis @dotnet Almost all the complaints about tracing are to do with either the inline impact of generation in general, or to do with not tail sampling and the cost/perf impact on the backend.

@RogerAlsing @dddanielreis @dotnet That's not how the buffers work in otel FWIW. I've seen more people complaining about the memory usage of the MeterProvider in .NET and Go than they have about spans since their memory impact is short lived when offload to the collector.

@RogerAlsing @dddanielreis @dotnet The data for metrics has to live around for the time of the aggregation. Meaning you're using allocations during that time. Depending on the overall cardinality of that metrics label that could be a lot. Spans however, are batched out every second.

@MartinDotNet @dddanielreis @dotnet "metrics require more resources in your app than pushing activities." In what way? Each individual metric is not a large chunk of data, and metrics do not require 100% of each measurement to be perfectly recorded. While spans contains much more metadata per instance.

@dddanielreis @RogerAlsing @dotnet You should work with the collector, either as a sidecar if you have to, or work with the platform team if not. Ideally though, you don’t even do, you generate the same graphs on the fly in the backend from the raw data.

@dddanielreis @RogerAlsing @dotnet That's an interesting point, but ultimately something that is a wider problem than generating telemetry. What you're doing by generating a metric is deciding what graph you want to display and how you'll use it. Which is ultimately wrong