(aws-lambda-event-sources): S3EventSource should take IBucket instead of Bucket

[guangyuzh]

Reproduction Steps

Cannot import an existing s3 bucket when creating Lambda s3 events:

1. aws_s3.Bucket.from_bucket_name(...) or aws_s3.Bucket.from_bucket_arn(...), etc. returns aws_s3.IBucket type
2. However, aws_lambda_event_sources.S3EventSource only takes aws_s3.Bucket type as a parameter
3. Note: aws_s3.Bucket implements aws_s3.IBucket

Error Log

File "/usr/local/lib/python3.6/site-packages/jsii/_runtime.py", line 66, in __call__
    inst = super().__call__(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/aws_cdk/aws_lambda_event_sources/__init__.py", line 199, in __init__
    jsii.create(S3EventSource, self, [bucket, props])
  File "/usr/local/lib/python3.6/site-packages/jsii/_kernel/__init__.py", line 208, in create
    overrides=overrides,
  File "/usr/local/lib/python3.6/site-packages/jsii/_kernel/providers/process.py", line 331, in create
    return self._process.send(request, CreateResponse)
  File "/usr/local/lib/python3.6/site-packages/jsii/_kernel/providers/process.py", line 316, in send
    raise JSIIError(resp.error) from JavaScriptError(resp.stack)
jsii.errors.JSIIError: Object of type @aws-cdk/aws-s3.IBucket is not convertible to @aws-cdk/aws-s3.Bucket

Environment

• CLI Version : v1.10.1
• Framework Version: v1.10.1
• OS : Mac OS
• Language : Python

Other

aws_lambda_event_sources.S3EventSource should take aws_s3.IBucket as a parameter's type.

---

This is 🐛 Bug Report

[AjkayAlan]

+1 - This is causing us some pain around unit testing S3EventSource in C#.

[leantorres73]

+1 - same issue for Typescript

[TonyFNZ]

The underlying issue here is that CloudFormation can only set the NotificationConfiguration on a S3 Bucket in the CFN template which created the bucket. This means that if an existing S3 bucket is imported into a CDK stack then it is not possible to update the notification configuration of the bucket, even if the parameter types on aws_lambda_event_sources.S3EventSource are changed.

Please +1 to get this changed in CFN aws-cloudformation/cloudformation-coverage-roadmap#79

[purnesh]

+1

[pigna90]

+1

[iliapolo]

@TonyFNZ Wrote:

The underlying issue here is that CloudFormation can only set the NotificationConfiguration on a S3 Bucket in the CFN template which created the bucket.

Yes, you can see more details here: #2004

Please +1 to get this changed in CFN aws-cloudformation/cloudformation-coverage-roadmap#79

Yes, please do that :) (rather than on this cdk issue).

[iliapolo]

Converting to a feature-request which is somewhat a duplicate of #2004.

[ThomasKiec]

@TonyFNZ I think there should be another way as well. Maybe some workaround in the background without Cloudformation.

The serverless team achieved it(See https://www.serverless.com/framework/docs/providers/aws/events/s3#using-existing-buckets). Then the @aws-cdk team should also find a way to solve this?

[iliapolo]

@ThomasKiec Yes - we actually have a solution in mind and probably won't be waiting on CFN to add this support. So stay tuned.

[ireneaguilar-seat]

Any update for this issue?

[itharavi]

+1. Facing the same issue in our case too.

[istvanszabo92]

Similar problem with 1.115.0 (Object of type @aws-cdk/aws-s3.BucketBase is not convertible to @aws-cdk/aws-s3.Bucket)

[farshadniayeshpour]

@eladb @iliapolo Unfortunately, this is something that forces me not to use CDK on a monthly basis even though I start writing my app in CDK but then realize we cannot create a lambda event source with an existing bucket. I would really appreciate it if you could resolve this or provide some type of recommendations so I can start working on it.

[nija-at]

@farshadniayeshpour - have you tried using the bucket notification mechanism in the s3 module - https://github.com/aws/aws-cdk/tree/master/packages/%40aws-cdk/aws-s3#bucket-notifications. This supports setting up notification on imported buckets to invoke a lambda function.

[LucasOliveiraS]

+1. Facing the same issue.

[kornicameister]

@nija-at using this module you can't exactly submit to objectcreated:*. You can choose one event type like objectcreated:post but if you try to submit another notification for, say, multipart it will tell you that you have two notifications with overlapping configuration (prefix, suffix). Perhaps that would be a workaround if there were ability to specify wildcard event?

Hope I didn't confuse things up ;P

[ThomasKiec]

@kornicameister Subscribing all object created events is possible. When adding the event notification via addEventNotification you can simply use the EventType.OBJECT_CREATED (=s3:ObjectCreated:*).

See @aws-cdk/aws-s3: EventType interface

A wildcard pattern is probably not useful, since currently only the event types listed here are supported by AWS. Also, the above interface contains all supported event types

[ericbn]

Since cdk 1.110.0, this is possible for existing buckets using IBucket.add_event_notification as @ThomasKiec mentioned.

Here's an example for reference:

import aws_cdk.aws_s3 as s3
import aws_cdk.aws_s3_notifications as s3n

bucket = s3.Bucket.from_bucket_name(scope, 'bucket-id', bucket_name='bucket-name')
bucket.add_event_notification(s3.EventType.OBJECT_CREATED, s3n.LambdaDestination(lambda_fn), prefix="prefix/")

The enhancement for this was merged in this PR: #15158

[deepak3082]

Does this fix also work for lambda.add_event_source(S3EventSource) . I am still getting an error with the following code block. I have this code in my create-lambda stack and i am on cdk 1.122.0 version.

myexistingbucket = _s3.Bucket.from_bucket_name(self, 'existingbucket', 'mybucketName')

lambda_trigger.add_event_source(S3EventSource(myexistingbucket,
                                                      events=[_s3.EventType.OBJECT_CREATED],
                                                      filters=[_s3.NotificationKeyFilter(prefix="lambdacode/")]
                                                      )
                                        )

[ericbn]

@deepak3082, not sure about versions after cdk 1.110.0. But up to that version, only IBucket.add_event_notification works. And it actually will create the exact CloudFormation structure as expected. You're just defining the event notification from the bucket to the lambda, instead of from the lambda to the bucket.

[deepak3082]

@ericbn .yes, my use case is to trigger lambda once a file is created/updated in s3 and i can not make any code change in the create-s3-stack. Hence, i want to update the create-lambda-stack code and add the eventsource method per aws cdk docs and I am not able to use IBucket.

[bchathoth]

+1. Facing same issue with cdk 1.130.0 version

[Xyz426]

+1, Any update? CDK V2 still can not use IBucket.

[alxojy]

any updates on this issue? 3 years later, still not fixed

[weijiany]

+1

[wcheek]

I still get an error when trying to define the event notification on the lambda function, but things work when defining on the bucket as suggested by @ericbn. So although this issue should still be fixed, it's no longer a breaking problem. Simply create an IBucket object using its name or arn and then add_event_notification()

[luca-ferreri]

In my case the solution of @ericbn proposed here worked only partially. In facts, with .add_event_notifications I got the error

AttributeError: '_BucketBaseProxy' object has no attribute 'add_event_notfication'

while I had no problem with .add_objcet_created_notification.

My working snippet follows for reference

bucket = aws_s3.Bucket.from_bucket_name(
    self, id="id-bucket", bucket_name="bucket-name"
)
bucket.add_object_created_notification(
    aws_s3_notifications.LambdaDestination(lambda_fn),
    aws_s3.NotificationKeyFilter(prefix="someprefix/"),
)

[Smikha]

Using aws_s3.Bucket.from_bucket_name(self, id='bucket-name', bucket_name="some_bucket_name")
Causes my lambda stacks to throw the following error:

Received response status [FAILED] from custom resource. Message returned: Error: An error occurred (NoSuchBucket) when calling the PutBucketNotificationConfiguration operation: The specified bucket does not exist.

Alternatively, using aws_s3.Bucket.from_bucket_arn(self, id='bucket-name',bucket_arn='some-bucket-arn') works fine.

Any hope for a fix?

[kornicameister]

I think from importing by name implies both account and region you are currently deploying to whereas the ARN version allows to explicitly set both of those.

[lkr2des]

+1

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.