Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(glue): s3 path specified in --spark-event-logs-path needs to end with slash #29357

Merged
merged 10 commits into from
Mar 19, 2024

Conversation

moomindani
Copy link
Contributor

Issue # (if applicable)

Closes #29356.

Reason for this change

Currently S3 path specified in --spark-event-logs-path does not end with slash in case only bucket is provided but prefix is not provided. This parameter causes errors when viewing the event log through Spark UI / Spark history server.

Description of changes

Add trailing slash when it does not end with slash.

Description of how you validated changes

Completed unit test and integ test.

Checklist


By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

@github-actions github-actions bot added repeat-contributor [Pilot] contributed between 3-5 PRs to the CDK bug This issue is a bug. p2 labels Mar 4, 2024
@aws-cdk-automation aws-cdk-automation requested a review from a team March 4, 2024 22:59
Copy link
Collaborator

@aws-cdk-automation aws-cdk-automation left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The pull request linter has failed. See the aws-cdk-automation comment below for failure reasons. If you believe this pull request should receive an exemption, please comment and provide a justification.

A comment requesting an exemption should contain the text Exemption Request. Additionally, if clarification is needed add Clarification Request to a comment.

Copy link
Contributor

@scanlonp scanlonp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @moomindani, this looks great! Have you confirmed that this works in Spark UI manually?

One thought: adding the trailing slash to the url seems like it could be used in more places than this. If so, we could add this functionality to the s3UrlForObject() function itself. Something along the lines of
public s3UrlForObject(key?: string, trailingSlash?: boolean): string {
in bucket.ts.

Then this line would be bucket.s3UrlForObject(props.prefix, true).

Lastly, this is changing current templates. Are we sure that

  1. This is not breaking
  2. if it is, that old templates were not working in the first place

We have some leeway since this is an alpha package, but we want to be clear on the PR if this is breaking.

@moomindani
Copy link
Contributor Author

Hi @scanlonp,
Thanks for quick feedback and comment

Have you confirmed that this works in Spark UI manually?

Yes.

Lastly, this is changing current templates. Are we sure that

  1. This is not breaking
  2. if it is, that old templates were not working in the first place

Yes, actually it introduces change to the template but as you noticed the old template without trailing slash did not work at all in all the places.

Let me try to make changes on bucket.s3UrlForObject then update this PR.

@moomindani moomindani changed the title fix(glue): S3 path specified in needs to end with slash fix(glue): s3 path specified in --spark-event-logs-path needs to end with slash Mar 7, 2024
@github-actions github-actions bot added the effort/medium Medium work item – several days of effort label Mar 7, 2024
@aws-cdk-automation aws-cdk-automation dismissed their stale review March 7, 2024 06:45

✅ Updated pull request passes all PRLinter validations. Dismissing previous PRLinter review.

@mergify mergify bot dismissed scanlonp’s stale review March 7, 2024 07:14

Pull request has been modified.

@aws-cdk-automation aws-cdk-automation added the pr/needs-community-review This PR needs a review from a Trusted Community Member or Core Team Member. label Mar 7, 2024
@scanlonp scanlonp self-assigned this Mar 8, 2024
@aws-cdk-automation
Copy link
Collaborator

This PR cannot be merged because it has conflicts. Please resolve them. The PR will be considered stale and closed if it remains in an unmergeable state.

@aws-cdk-automation aws-cdk-automation removed the pr/needs-community-review This PR needs a review from a Trusted Community Member or Core Team Member. label Mar 11, 2024
@aws-cdk-automation aws-cdk-automation added the pr/needs-community-review This PR needs a review from a Trusted Community Member or Core Team Member. label Mar 12, 2024
@scanlonp
Copy link
Contributor

Hey @moomindani, I fear I may have led you down the wrong path. I talked with some other team members, and my suggestion to change the function in the s3 package was the wrong call. We should keep this change scoped to glue. I feel pretty good approving your original PR if you would not mind going back to your first commit. Maybe adding a small comment on the purpose of the regex, if you would like.

Apologies again, but hope we can get this in smoothly from here!

Copy link
Collaborator

@aws-cdk-automation aws-cdk-automation left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The pull request linter has failed. See the aws-cdk-automation comment below for failure reasons. If you believe this pull request should receive an exemption, please comment and provide a justification.

A comment requesting an exemption should contain the text Exemption Request. Additionally, if clarification is needed add Clarification Request to a comment.

@aws-cdk-automation aws-cdk-automation removed the pr/needs-community-review This PR needs a review from a Trusted Community Member or Core Team Member. label Mar 13, 2024
@aws-cdk-automation aws-cdk-automation dismissed their stale review March 13, 2024 05:35

✅ Updated pull request passes all PRLinter validations. Dismissing previous PRLinter review.

@aws-cdk-automation aws-cdk-automation added the pr/needs-community-review This PR needs a review from a Trusted Community Member or Core Team Member. label Mar 13, 2024
@moomindani
Copy link
Contributor Author

@scanlonp Thanks for your comment, I understand it.
I fixed the PR with reverting S3 side change. Can you take another look?

scanlonp
scanlonp previously approved these changes Mar 19, 2024
@scanlonp scanlonp removed the pr/needs-community-review This PR needs a review from a Trusted Community Member or Core Team Member. label Mar 19, 2024
Copy link
Contributor

mergify bot commented Mar 19, 2024

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

@mergify mergify bot dismissed scanlonp’s stale review March 19, 2024 06:43

Pull request has been modified.

@aws-cdk-automation
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: AutoBuildv2Project1C6BFA3F-wQm2hXv2jqQv
  • Commit ID: 5cb1ef2
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

Copy link
Contributor

mergify bot commented Mar 19, 2024

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

@mergify mergify bot merged commit 4ff3565 into aws:main Mar 19, 2024
11 checks passed
ahammond pushed a commit to ahammond/aws-cdk that referenced this pull request Mar 26, 2024
…with slash (aws#29357)

### Issue # (if applicable)

Closes aws#29356.

### Reason for this change

Currently S3 path specified in `--spark-event-logs-path` does not end with slash in case only bucket is provided but prefix is not provided. This parameter causes errors when viewing the event log through Spark UI / Spark history server.

### Description of changes

Add trailing slash when it does not end with slash.

### Description of how you validated changes

Completed unit test and integ test.

### Checklist
- [x] My code adheres to the [CONTRIBUTING GUIDE](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md) and [DESIGN GUIDELINES](https://github.com/aws/aws-cdk/blob/main/docs/DESIGN_GUIDELINES.md)

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug This issue is a bug. effort/medium Medium work item – several days of effort p2 repeat-contributor [Pilot] contributed between 3-5 PRs to the CDK
Projects
None yet
Development

Successfully merging this pull request may close these issues.

(aws-glue-alpha): S3 path specified in --spark-event-logs-path needs to end with slash
3 participants