-
Notifications
You must be signed in to change notification settings - Fork 2.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
New component: RabbitMQ Exporter #28891
Comments
Sounds like a valid use case and component to me! Please review all the requirements of adding a new component. From here you'll need a sponsor to be able to move forward. You can join the Collector sig meetings and add this issue to the agenda of the Google doc to get more attention here. As shared in another component proposal, not all components end up being sponsored, so feel free to go ahead and implement this in your own repository if you're not able to get much traction here soon. Thanks for the proposal and willingness to contribute! |
Confirm this is not a vendor-specific component, from what I can tell. |
How much of this exporter would you be able to calc after the kafka exporter? Ideally it could be a thin client and use encoding extensions to do most of the heavy lifting, making maintenance easier. |
Thanks @crobert-1 and @atoulme. I was assuming this wouldn't get sponsored and started implementing it just as a learning exercise. Here's what I have so far. This implementation might have optimizations and configuration options that aren't worth the complexity for an alpha component though. I'll list out the possible scope to help decide if it's worth sponsoring/maintaining. From there I could break the implementation into smaller tasks / pull requests. Lmk what you suggest Possible MVP
Other possible enhancements
|
This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping |
@swar8080 I'd be happy to sponsor this component and help it land in contrib. |
**Description:** Sets-up the configuration format and common component boilerplate for the rabbitmq exporter Implementation will be in other pull requests **Link to tracking Issue:** [28891](#28891) **Testing:** Standard initial unit tests **Documentation:** Created README for the component --------- Co-authored-by: Antoine Toulme <antoine@toulme.name>
Is this the Exporter Part of #10592? The existing RabbitMQ receiver can't be used to receive the exported data, right or am I missing something? |
@romerod yep, same idea as the issue you mentioned. This component is for sending telemetry to rabbitmq. The rabbitmqreceiver is for collecting rabbitmq usage metrics |
Thanks @swar8080, understood this, but which receiver can be used to receive the telemetry on the receiving side? |
@romerod gotcha, so there's no component for pulling messages from rabbitmq. This component is just for pushing messages to rabbitmq |
**Description:** This is the completed implementation of the rabbitmq exporter. **Link to tracking Issue:** #28891 **Testing:** - Unit tests - Happy path with rabbitmq running locally and in the cloud, testing different configuration options - Error cases - Fail to connect during start-up - Invalid credentials - Connection lost midway through publishing to the queue. The component attempts reconnecting on the next publish attempt - Concurrent publishing, both with and without connection issues **Documentation:** Updated README with more configuration options --------- Co-authored-by: Andrzej Stencel <astencel@sumologic.com>
**Description:** Sets-up the configuration format and common component boilerplate for the rabbitmq exporter Implementation will be in other pull requests **Link to tracking Issue:** [28891](open-telemetry#28891) **Testing:** Standard initial unit tests **Documentation:** Created README for the component --------- Co-authored-by: Antoine Toulme <antoine@toulme.name>
**Description:** This is the completed implementation of the rabbitmq exporter. **Link to tracking Issue:** open-telemetry#28891 **Testing:** - Unit tests - Happy path with rabbitmq running locally and in the cloud, testing different configuration options - Error cases - Fail to connect during start-up - Invalid credentials - Connection lost midway through publishing to the queue. The component attempts reconnecting on the next publish attempt - Concurrent publishing, both with and without connection issues **Documentation:** Updated README with more configuration options --------- Co-authored-by: Andrzej Stencel <astencel@sumologic.com>
This is an exciting component! |
**Description:** <Describe what has changed.> <!--Ex. Fixing a bug - Describe the bug and how this fixes the issue. Ex. Adding a feature - Explain what this achieves.--> @atoulme [volunteered to be the sponsor](#28891 (comment)) of this component, so I believe he should be listed as a code owner. From [CONTRIBUTING.md](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/CONTRIBUTING.md#adding-new-components): ``` A sponsor is an approver who will be in charge of being the official reviewer of the code and become a code owner for the component. ``` **Link to tracking Issue:** <Issue number if applicable> #28891
**Description:** <Describe what has changed.> <!--Ex. Fixing a bug - Describe the bug and how this fixes the issue. Ex. Adding a feature - Explain what this achieves.--> @atoulme [volunteered to be the sponsor](open-telemetry#28891 (comment)) of this component, so I believe he should be listed as a code owner. From [CONTRIBUTING.md](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/CONTRIBUTING.md#adding-new-components): ``` A sponsor is an approver who will be in charge of being the official reviewer of the code and become a code owner for the component. ``` **Link to tracking Issue:** <Issue number if applicable> open-telemetry#28891
Overiew
Use-cases
Similar use cases as other durable messaging system exporters like Kafka and Pulsar. This could help meet teams where they are that want to do custom telemetry processing but only have access to RabbitMQ
Why prioritize RabbitMQ and not some other queue?
Other options considered
SQS
SQS is also popular but i'm assuming someone from AWS would need to own/maintain the component
AMQP 1.0 Protocol
This would allow supporting other queues like ActiveMQ with a Go client library maintained by microsoft. However:
STOMP Protocol
STOMP would also allow supporting other queues like ActiveMQ, however:
JMS
A JMS exporter was requested in #27258 because many queues support it
However, JMS is an API and not a wire-protocol. The implementation of the protocol seems specific to each queue
Example configuration for the component
Unique Configuration
publisher
routing_key
(default = otlp_spans for traces, otlp_metrics for metrics, otlp_logs for logs): The AMQP routing key for the message, which will be delivered to a queue with that name (using the amq.direct exchange)confirm_mode
(default = true) whether to wait for confirmation that RabbitMQ successfully received or is unable to process a message. This improves the accuracy of collector metrics on unprocessed data. The tradeoff is lower throughput having to wait for asynchronous confirmation.durable
(default = true) whether to instruct RabbitMQ to durably persist messages on disk. Whenpublisher.confirm_mode
istrue
, this may delay confirmation by a few hundred milliseconds , decreasing the pipeline's throughput.endpoint
(default =rabbit://localhost:5672): The url of the RabbitMQ broker.Common Configuration
The below is copied from the Pulsar exporter since it seems relevant to this exporter as well
auth
/tls
settings (TODO)encoding
of messages set to RabbitMQ (TODO, need to research current OTEL best-practices)timeout
: timeout for sending an individual messageconnection_timeout
: timeout for the establishing a connection to the broker and creating an AMQP channelretry_on_failure
enabled
:initial_interval
: Time to wait after the first failure before retrying; ignored ifenabled
isfalse
max_interval
(default = ?): Is the upper bound on backoff; ignored ifenabled
isfalse
max_elapsed_time
(default = ?): Is the maximum amount of time spent trying to send a batch; ignored ifenabled
isfalse
sending_queue
enabled
(default = true)num_consumers
: Number of consumers that dequeue batches; ignored ifenabled
isfalse
queue_size
Maximum number of batches kept in memory before dropping data; ignored ifenabled
isfalse
; User should calculate this asnum_seconds * requests_per_second
where:num_seconds
is the number of seconds to buffer in case of a backend outagerequests_per_second
is the average number of requests per seconds.Telemetry data types supported
Logs, metrics, and traces
Is this a vendor-specific component?
Code Owner(s)
@swar8080
Sponsor (optional)
@atoulme
Additional context
Is this considered a vendor-specific component that needs to be implemented/maintained by the RabbitMQ team? If it's not then i'm happy to implement this!
There's some more research needed for the design but i'll wait to see if this accepted/sponsored before going down that rabbit hole. Let me know if any extra info would help though
The text was updated successfully, but these errors were encountered: