Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New component: Cloudwatch metric streams encoding extension #37870

Closed
constanca-m opened this issue Feb 12, 2025 · 7 comments · Fixed by open-telemetry/opentelemetry-collector-releases#883
Labels
Accepted Component New component has been sponsored

Comments

@constanca-m
Copy link
Contributor

constanca-m commented Feb 12, 2025

The purpose and use-cases of the new component

Some aws receivers and exporters, like awsfirehosereceiver, get data from cloudwatch metric streams. The purpose of this extension is to transform these metrics into otel metrics.

Relates to #37113.

Example configuration for the component

extensions:
  cloudwatch_metric_streams_encoding:
     # See output format: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-cloudwatch-metricstream.html#cfn-cloudwatch-metricstream-outputformat
     # format: opentelemetry1.0 # options [json, opentelemetry1.0], and opentelemetry0.7 not supported.

receivers:
  awsfirehose:
    include_metadata: true
    access_key: "access_key"
    endpoint: :4434
    encoding: cloudwatch_metric_streams_encoding

Telemetry data types supported

  • Metrics

Code Owner(s)

@axw

Sponsor (optional)

@atoulme

Additional context

No response

@constanca-m constanca-m added needs triage New item requiring triage Sponsor Needed New component seeking sponsor labels Feb 12, 2025
@axw
Copy link
Contributor

axw commented Feb 12, 2025

Adding to the motivation slightly: the CloudWatch formats (logs & metrics) understood by the awsirehosereceiver could also be used in other receivers, e.g. the S3 receiver, which already supports the encoding extension.

@constanca-m and I discussed offline: my understanding is that since she is not (yet) a member of the OpenTelemetry org, she cannot be a code owner at this time. I'm happy to take on code ownership, and later on I would ideally add @constanca-m as a code owner too.

@Aneurysm9
Copy link
Member

As cloudwatch describes a lot of things I think that any encoding extensions that process data produced by CloudWatch systems should be explicit in their naming about the type of data they process. For instance, CloudWatch Metric Streams can produce three different output formats (though two of those are different OTLP versions). This is different from EMF, which is another CloudWatch metric encoding scheme that may be found in, for example, log records or S3 objects.

@constanca-m constanca-m changed the title New component: Cloudwatch encoding extension New component: Cloudwatch metric streams encoding extension Feb 27, 2025
@constanca-m
Copy link
Contributor Author

constanca-m commented Feb 27, 2025

For instance, CloudWatch Metric Streams can produce three different output formats

Thanks @Aneurysm9, I have updated the issue to only be applicable to cloudwatch metric streams for now. In the future, we should add encodings for cloudwatch logs and the other types of CW metrics.

@atoulme
Copy link
Contributor

atoulme commented Mar 5, 2025

I can sponsor this component.

@atoulme atoulme added Accepted Component New component has been sponsored and removed Sponsor Needed New component seeking sponsor labels Mar 5, 2025
@axw
Copy link
Contributor

axw commented Mar 5, 2025

Initial skeleton PR: #38388

atoulme pushed a commit that referenced this issue Mar 5, 2025

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
#### Description

Create the component skeleton and basic config tests. Unmarshallers will
be implemented in a follow up PR.

#### Link to tracking issue

First part of
#37870

#### Testing

Added basic unit tests.

#### Documentation

Added README.
@atoulme atoulme removed the needs triage New item requiring triage label Mar 8, 2025
@axw
Copy link
Contributor

axw commented Mar 11, 2025

#38407 is merged. Once #38408 is done I think we can look at progressing this to Alpha stability.

@axw
Copy link
Contributor

axw commented Mar 18, 2025

#38704 is up to bump to Alpha.

atoulme pushed a commit that referenced this issue Mar 18, 2025
…ha (#38704)

#### Description

Bump AWS CloudWatch Metric Streams encoding extension to Alpha
stability.
The component is fully functional, has comprehensive unit tests, and
documentation.

#### Link to tracking issue

Next step for #37870

#### Testing

N/A

#### Documentation

N/A
songy23 pushed a commit that referenced this issue Mar 19, 2025

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
…ers (#38789)

<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue.
Ex. Adding a feature - Explain what this achieves.-->
#### Description

Update codeowners.

Work done:
- PRs reviewed / authored:
-
#38419
-
#38516
-
#38605
- Issues responded to:
-
#37870

<!-- Issue number (e.g. #1234) or full URL to issue, if applicable. -->
#### Link to tracking issue
N/A


<!--Describe what testing was performed and which tests were added.-->
#### Testing

N/A

<!--Describe the documentation added.-->
#### Documentation

N/A
lightme16 pushed a commit to lightme16/opentelemetry-collector-contrib that referenced this issue Mar 19, 2025

Verified

This commit was signed with the committer’s verified signature.
msclock l.feng
…ers (open-telemetry#38789)

<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue.
Ex. Adding a feature - Explain what this achieves.-->
#### Description

Update codeowners.

Work done:
- PRs reviewed / authored:
-
open-telemetry#38419
-
open-telemetry#38516
-
open-telemetry#38605
- Issues responded to:
-
open-telemetry#37870

<!-- Issue number (e.g. open-telemetry#1234) or full URL to issue, if applicable. -->
#### Link to tracking issue
N/A


<!--Describe what testing was performed and which tests were added.-->
#### Testing

N/A

<!--Describe the documentation added.-->
#### Documentation

N/A
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Accepted Component New component has been sponsored
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants