Output trace.id and span.id in log format.

[q42jaap]

Since serilog/serilog#1955 is implemented by Serilog I want to propagate trace id and span id from LogEvent to the newrelic log format.

I think NewRelicFormatter could be updated to use LogEvent.TraceId and LogEvent.SpanId. This would also mean upgrading to serilog 3 probably.

[workato-integration]

https://new-relic.atlassian.net/browse/NR-264778

[nr-ahemsath]

Hi @q42jaap, thank you for your feature request submission.

So, the trace id and span id in question here are the OpenTelemetry trace id and span id. The log enricher library in this repo is meant for enriching log data with New Relic proprietary data to help link log events to New Relic APM entities (.NET applications, in this case). Adding the OpenTelemetry trace/span id to log events from this library wouldn't be useful to help link those log events to any New Relic APM entities.

Are you using Open Telemetry to monitor other applications with New Relic?

[q42jaap]

Hi @nr-ahemsath. We have a k8s setup where the standard output of the logs are forwarded to NewRelic by an agent that is installed on the node. The k8s pod that we maintain does not forward logs itself.

Metrics are uploaded via an OTEL endpoint, but logs aren't.

So, for us it makes sense to use the console output together with the NewRelicFormatter, but for the integration in NR we need trace.id and span.id as well as service.name to be exposed in the log format.

[nrcventura]

The log enricher plugins in this repo are not designed to support OpenTelemetry data. Since you are collecting metrics using OpenTelemetry, I assume that you are also using OpenTelemetry to collect traces. If that's the case you should also use OpenTelemetry to collect logs. By using OTel to collect all 3 signals, it allows for the maximum correlation of data between the different signal types. If both your traces and logs are sent to New Relic, you should be able to see this data linked together within the New Relic UI.

As a workaround, if you would prefer to use the log enrichers from this repo, you will likely need to leverage the features of your logging library to enrich your data. For example, with serilog you can specify the enrichers to use to add additional attributes to your log events, and you can add your own enrichers to add that data to the log events, and then just use the formatter in this plugin. This workaround may not enable New Relic's logs in context functionality.

The plugins in this repo are designed to handle two use cases.

1. Allowing application logs to be formatted in a more structured manner before sending them to the configured destination
2. Providing contextual information from New Relic agents and then structuring the logs before sending them to the configured destination

[q42jaap]

@nrcventura Serilog added TraceId and SpanId to their LogEvent which NewRelicFormatter is formatting. It would make sense that a formatter for serilog uses these and formats them.
We're in an environment where filebeat is picking up logs and putting them into new relic, hence the need to format service.name, trace.id and span.id.

I don't see the option in NewRelicFormatter to allow for user properties to format to exactly trace.id or span.id. So the NewRelicFormatter would not be usable in this specific use-case?

[nrcventura]

@q42jaap I agree that the NewRelicFormatter is not intended to solve your particular use case.

Yes, Serilog supports trace id and span id, but those trace ids and span ids do not match the same trace ids and span ids that the New Relic .net agent is generating, because the trace ids and span ids that Serilog is collecting are the trace ids and span ids from OpenTelemetry. The New Relic .net agent is not an OpenTelemetry agent, and the log enrichers in this repo are primarily intended to work with the New Relic .net agent.

To support what you want, you really just need a json formatted output that enriches your log events with trace.id, span.id, and service.name. The service.name attribute is what's used to associate the log data to an instrumented OpenTelemetry service so that the APM UI can show the relevant logs for that application. The trace.id and span.id are used to link the trace data to the logs for that trace. The NewRelic formatter in this repo is really just a json formatter that tries to standardize some naming conventions for some of the json properties, and treats the enriched data from the New Relic .net agent as a special collection of properties. You can probably just configure the Serilog json formatter to do something similar since you do not have any data from the New Relic .net agent in your enriched log event.

[q42jaap]

@nrcventura Thanks for your explanation and guidance. I'm going to close the issue 🙏