-
Notifications
You must be signed in to change notification settings - Fork 1.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Introduce new exporter helper with batching option #8122
Comments
Introduce a new exporter helper that operates over client-provided requests instead of pdata. The helper user now has to provide `Converter` - an interface with a function implementing translation of pdata Metrics/Traces/Logs into a user-defined `Request`. `Request` is an interface with only one required function `Export`. It opens a door for moving batching to the exporter, where batches will be built from client data format, instead of pdata. The batches can be properly sized by custom request size, which can be different from OTLP. The same custom request sizing will be applied to the sending queue. It will also improve the performance of the sending queue retries for non-OTLP exporters, they don't need to translate pdata on every retry. This is an implementation alternative to #7874 as suggested in #7874 (comment) Tracking Issue: #8122 --------- Co-authored-by: Alex Boten <alex@boten.ca>
This change adds collector's internal metrics and tracing to the new request-based exporter helpers. Only those metrics and traces are added that are already adopted by the existing exporter helpers for backward compatibility. The new exporter helpers can and should expose more metrics in the future, e.g. for tracking converter errors. Tracking Issue: #8122
Some feedback about the interface:
|
Thanks for the feedback!
That's what I wanted to do after #8248 is merged 👍 |
…on (#8764) As proposed in #8122 (comment) If we need backward conversion, we will use an optional argument to the helper function instead of an optional interface.
#9164) Introduce an option to limit the queue size by the number of items instead of number of requests. This is preliminary step for having the exporter helper v2 with a batcher sender placed after the queue sender. Otherwise, it'll be hard for the users to estimate the queue size based on the number of requests without batch processor in front of it. This change doesn't effect the existing functionality and the items based queue limiting cannot be utilized yet. Updates #8122 Alternative to #9147
Introduce a way to enable queue in the new exporter helper with a developer interface suggested in #8248 (comment). The new configuration interface for the end users provides a new `queue_size_items` option to limit the queue by a number of spans, log records, or metric data points. The previous way to limit the queue by number of requests is preserved under the same field, `queue_size,` which will later be deprecated through a longer transition process. Tracking issue: #8122
This change introduces new experimental batching functionality to the exporter helper. The batch sender is fully concurrent and synchronous. It's set after the queue sender, which, if enabled, introduces the asynchronous behavior and ensures no data loss with the permanent queue. Follow-up TODO list: - Add pre-built merge funcs for pdata - Handle partial errors - A missing part compared to the batch processor is the ability to shard the batches by context value. Updates #8122
Introduce default batching functionality based on the internal data type (pdata). This makes the exporter batching capability available to the regular exporter helpers without using custom requests. Updates #8122
@verejoel, the batch sharding will be added for parity with the batch processor. But there are no plans to have separate sender pipelines. The sharding will ensure that one batch won't have data with different metadata values but the batches will be processed by one pipeline. |
To properly support multi-tenant pipeline, having a sender queue per batch will be critical. Otherwise, we can still have batches from tenants with back-pressure blocking the pipeline for all tenants attempting to ship data. Can we work in sharded queues as part of this feature? Or should I write it down as a separate feature request? |
@dmitryax Thanks for the batch sender! I've been experimenting with it lately. In my experiments, the batch sender is producing inconsistent batch sizes which could be lower than desired due to goroutine scheduling even after #9761 . The scenario I usually run into is that given queue sender concurrency = batch sender concurrency limit = N, and they are all blocked on I tried to fix it by resetting On the performance side of things, I haven't benchmarked it, but I'm not sure how performant it is when queue consumer concurrency is high (e.g. 1000+) given the number of running goroutines. I understand that spawning goroutines and blocking them may be the only way to avoid deleting the entry in persistent queue before request export. I wish I could give some useful suggestions here, but I don't have a good idea within the current paradigm. Have you considered the limitations I mentioned, and are there bigger changes planned ahead? edit: I'm starting to think, will it be fine if the batch sender waits up to FlushTimeout even when concurrency limit is reached when not all goroutines are in the same batch? |
batch size & concurrency
I've drafted a PR to do so, and feedback is welcome: #9891 The issue is captured in #9952 batching based on bytes
I see that batching based on size rather than item count is planned but apparently not implemented yet. Is that still on the roadmap? |
The counting of batch sizes in bytes is something that would be super useful for us, since some of our backends limit us on payload size in bytes. @dmitryax let me know if there's anything I can do to help push that forward, would be happy to help out. |
This is a tracking issue for introducing the new exporter helper and migrating the existing exporters to use it.
The primary reason for introducing the new exporter helper is to move the batching to the exporter side and deprecate the batch processor as part of making the delivery pipeline reliable, as reported in #7460. More details about moving batching to the exporter helper can be found in #4646.
Shifting batching to the exporter side grants us the opportunity to leverage the exporter's data model instead of relying on OTLP. As a result, we can achieve the following benefits:
Adapting to the new exporter helper requires exporter developers to implement at least two functions:
Design document: https://docs.google.com/document/d/1uxnn5rMHhCBLP1s8K0Pg_1mAs4gCeny8OWaYvWcuibs
Sub-issues:
The text was updated successfully, but these errors were encountered: