-
Notifications
You must be signed in to change notification settings - Fork 1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Log Record
s passed to log processor OnEmit
is non-mutable
#5219
Comments
Record
s passed to log processors' OnEmit
need to mutableRecord
s passed to log processor OnEmit
is non-mutable
One solution is to have OnEmit(ctx context.Context, record *Record) error This will mean that every record will need to be allocated to the heap. |
Another solution is to have OnEmit(ctx context.Context, record Record) (Record, error) The returned |
I could work on this but it would be nice to have #5093 merged first to reduce the amount of conflicts. |
Merged |
I remember what was my idea. It was to use Processor as decorators in order to make modifications. A processor can modify the received record before passing to the next processor: type mutatingProcessor struct {
wrapped log.Processor
}
func (mutatingProcessor p) OnEmit(ctx context.Context, record Record) error {
r.SetBody(log.StringValue("REDACTED"))
return p.wrapped.OnEmit(ctx, r.Clone())
} Thanks to it you can have even two "processor" pipelines which do not interfere with each other. So you can have one processor pipeline which e.g. has a processor redacts secrets and wraps a simple processor with stdout exporter and then a second pipeline which uses a batch processor (no redacting) with OTLP exporter: log.NewLoggerProvider(
log.WithProcessor(mutatingProcessor{ log.NewSimpleProcessor(stdoutExp) }), // stdout exporter with simple processor, the log records' body are redacted
log.WithProcessor(log.NewBatchProcessor(otlpExp)), // otlp exporter with a batch processor
) @MrAlias, can we close this issue? I do not see how this implementation would break the specification and I think it is a safer and more usable design. |
No, I don't think we can close the issue. We are not compatible with the specification. The specification states the log records are mutable and that processors are executed in order. The processors after a mutation should receive the mutated record. |
Where is it defined? What do you think about updating the specification? |
In the definition of a read/write record. |
I think it is "in spirit" of the specification. I would rather ask the Specification SIG if it is an acceptable design and clarify the specification. |
It looks that the TC wants to have a possibility so that the mutated record will be passed to next registered processor without forcing the user to make a decorator. See: https://github.com/open-telemetry/opentelemetry-specification/pull/4062/files#r1619504122 PS. There is a chance that making I created #5448 that can help us comparing performance consequences between different Processor APIs such us:
|
Towards #5219 Towards #5054 This benchmarks are supposed - Validate the the `Processor` interface design from performance perspective. E.g. they are used to check if a processor that is modifying a log record is causing an additional heap allocations. - Benchmark the processors supported by the SDK. These are "almost-end-to-end" benchmarks (with noopExporter) so that it checks the performance of the SDK log processing without the actual exporting part. ``` cpu: Intel(R) Core(TM) i9-10885H CPU @ 2.40GHz BenchmarkProcessor/Simple-16 1990946 644.6 ns/op 417 B/op 1 allocs/op BenchmarkProcessor/Batch-16 835135 1211 ns/op 597 B/op 0 allocs/op BenchmarkProcessor/ModifyTimestampSimple-16 1782510 644.3 ns/op 417 B/op 1 allocs/op BenchmarkProcessor/ModifyTimestampBatch-16 945699 1222 ns/op 637 B/op 0 allocs/op BenchmarkProcessor/ModifyAttributesSimple-16 1570214 717.6 ns/op 465 B/op 2 allocs/op BenchmarkProcessor/ModifyAttributesBatch-16 768399 1383 ns/op 653 B/op 1 allocs/op ```
Blocked by:
Description
The specification requires that log processors be provided a
LogRecordProcessor
:Our current implementation Defines the
Processor
interface as such:Any modification made to the
record
passed to a processor'sOnEmit
method will not be applied to the subsequent processor. The record is not mutable, it does not comply with the specification requirement.The text was updated successfully, but these errors were encountered: