Unable to generate lambda from cross account s3 bucket

[ArendAMZN]

Hello, I'm trying to define the following setup in CDK

• s3Stack: defined in account A and it generates an s3 bucket
• lambdaStack: defined in account B and it generates a lambda using code from the bucket in s3Stack

I'm able to generate the s3Stack fine, but when I try to generate the lambdaStack it's unable to reference the s3 bucket cross account

Is this workflow possible in cdk?

Reproduction Steps

Code:

import lambda = require('@aws-cdk/aws-lambda');
import s3 = require('@aws-cdk/aws-s3');
import iam = require('@aws-cdk/aws-iam');
import kms = require('@aws-cdk/aws-kms');
import { RemovalPolicy, App, Stack, StackProps } from '@aws-cdk/core';
 
const app = new App();
 
const devEnv = {region: 'us-west-2', account: '363575151959'}
const toolEnv = {region: 'us-west-2', account: '776362166164'}
 
const s3Stack = new Stack(app, 's3Stack', { env: devEnv });
const s3Bucket = new s3.Bucket(s3Stack, 'bucket', {
  bucketName: 'castaren-s3-crossaccount-testing',
  removalPolicy: RemovalPolicy.DESTROY,
  encryption: s3.BucketEncryption.KMS,
  encryptionKey: new kms.Key(s3Stack, 'CrossAccountKmsKey',
 {
    removalPolicy: RemovalPolicy.DESTROY,
  }),
});
 
const s3Role = new iam.Role(s3Stack, 'CrossAccountRole', {
  roleName: 's3CrossAccountRoleName',
  assumedBy: new iam.AccountPrincipal(toolEnv['account']),
});
 
var s3CrossPolicy = new iam.PolicyStatement();
s3CrossPolicy.addAllResources();
s3CrossPolicy.addActions('s3:*');
s3Role.addToPolicy(s3CrossPolicy);
 
const lambdaStack = new Stack(app, 'lambdaStack', { env: toolEnv });
 
const lambdaFunction = new lambda.Function(lambdaStack, 'function ', {
  runtime: lambda.Runtime.PYTHON_3_7,
  handler: 'index.handler',
  code: lambda.Code.fromBucket(s3Bucket,'BETA'),
});
 
var funcCrossPolicy = new iam.PolicyStatement()
funcCrossPolicy.addResources(s3Role.roleArn);
funcCrossPolicy.addActions('sts:AssumeRole');
lambdaFunction.addToRolePolicy(funcCrossPolicy);

Commands

npm run build
cdk deploy s3Stack
cdk deploy lambdaStack

Error log below is from lambdaStack deployment

Error Log

00:06:56 castaren-> cdk deploy lambdaStack                                                                                                                                           
lambdaStack: deploying...
lambdaStack: creating CloudFormation changeset...
 0/5 | 4:44:15 PM | CREATE_IN_PROGRESS   | AWS::IAM::Role        | function /ServiceRole (functionServiceRoleD810D9B4) 
 0/5 | 4:44:15 PM | CREATE_IN_PROGRESS   | AWS::CDK::Metadata    | CDKMetadata 
 0/5 | 4:44:16 PM | CREATE_IN_PROGRESS   | AWS::IAM::Role        | function /ServiceRole (functionServiceRoleD810D9B4) Resource creation Initiated
 0/5 | 4:44:17 PM | CREATE_IN_PROGRESS   | AWS::CDK::Metadata    | CDKMetadata Resource creation Initiated
 1/5 | 4:44:17 PM | CREATE_COMPLETE      | AWS::CDK::Metadata    | CDKMetadata 
 2/5 | 4:44:34 PM | CREATE_COMPLETE      | AWS::IAM::Role        | function /ServiceRole (functionServiceRoleD810D9B4) 
 2/5 | 4:44:36 PM | CREATE_IN_PROGRESS   | AWS::IAM::Policy      | function /ServiceRole/DefaultPolicy (functionServiceRoleDefaultPolicy618511A6) 
 2/5 | 4:44:37 PM | CREATE_IN_PROGRESS   | AWS::IAM::Policy      | function /ServiceRole/DefaultPolicy (functionServiceRoleDefaultPolicy618511A6) Resource creation Initiated
 3/5 | 4:44:46 PM | CREATE_COMPLETE      | AWS::IAM::Policy      | function /ServiceRole/DefaultPolicy (functionServiceRoleDefaultPolicy618511A6) 
 3/5 | 4:44:48 PM | CREATE_IN_PROGRESS   | AWS::Lambda::Function | function  (function7F8023C1) 
 4/5 | 4:44:48 PM | CREATE_FAILED        | AWS::Lambda::Function | function  (function7F8023C1) Your access has been denied by S3, please make sure your request credentials have permission to GetObject for castaren-s3-crossaccount-testing/BETA. S3 Error Code: AccessDenied. S3 Error Message: Access Denied (Service: AWSLambdaInternal; Status Code: 403; Error Code: AccessDeniedException; Request ID: 059e22ba-ab44-44d0-9791-7ceca24faec1)
        new Function (/workspace/castaren/SearchResourceManager_PoC/node_modules/@aws-cdk/aws-lambda/lib/function.ts:436:35)
        \_ Object.<anonymous> (/workspace/castaren/SearchResourceManager_PoC/bin/search_resource_manager_poc.ts:104:24)
        \_ Module._compile (internal/modules/cjs/loader.js:778:30)
        \_ Module.m._compile (/workspace/castaren/SearchResourceManager_PoC/node_modules/ts-node/src/index.ts:493:23)
        \_ Module._extensions..js (internal/modules/cjs/loader.js:789:10)
        \_ Object.require.extensions.(anonymous function) [as .ts] (/workspace/castaren/SearchResourceManager_PoC/node_modules/ts-node/src/index.ts:496:12)
        \_ Module.load (internal/modules/cjs/loader.js:653:32)
        \_ tryModuleLoad (internal/modules/cjs/loader.js:593:12)
        \_ Function.Module._load (internal/modules/cjs/loader.js:585:3)
        \_ Function.Module.runMain (internal/modules/cjs/loader.js:831:12)
        \_ Object.<anonymous> (/workspace/castaren/SearchResourceManager_PoC/node_modules/ts-node/src/bin.ts:158:12)
        \_ Module._compile (internal/modules/cjs/loader.js:778:30)
        \_ Object.Module._extensions..js (internal/modules/cjs/loader.js:789:10)
        \_ Module.load (internal/modules/cjs/loader.js:653:32)
        \_ tryModuleLoad (internal/modules/cjs/loader.js:593:12)
        \_ Function.Module._load (internal/modules/cjs/loader.js:585:3)
        \_ Function.Module.runMain (internal/modules/cjs/loader.js:831:12)
        \_ findNodeScript.then.existing (/usr/lib/node_modules/npm/node_modules/libnpx/index.js:268:14)
 4/5 | 4:44:49 PM | ROLLBACK_IN_PROGRESS | AWS::CloudFormation::Stack | lambdaStack The following resource(s) failed to create: [function7F8023C1]. . Rollback requested by user.
 4/5 | 4:45:03 PM | DELETE_IN_PROGRESS   | AWS::CDK::Metadata    | CDKMetadata 
 5/5 | 4:45:03 PM | DELETE_COMPLETE      | AWS::Lambda::Function | function  (function7F8023C1) 
 5/5 | 4:45:04 PM | DELETE_IN_PROGRESS   | AWS::IAM::Policy      | function /ServiceRole/DefaultPolicy (functionServiceRoleDefaultPolicy618511A6) 
 6/5 | 4:45:05 PM | DELETE_COMPLETE      | AWS::CDK::Metadata    | CDKMetadata 
 7/5 | 4:45:05 PM | DELETE_COMPLETE      | AWS::IAM::Policy      | function /ServiceRole/DefaultPolicy (functionServiceRoleDefaultPolicy618511A6) 
 7/5 | 4:45:06 PM | DELETE_IN_PROGRESS   | AWS::IAM::Role        | function /ServiceRole (functionServiceRoleD810D9B4) 
 
 ❌  lambdaStack failed: Error: The stack named lambdaStack failed creation, it may need to be manually deleted from the AWS console: ROLLBACK_COMPLETE
 8/5 | 4:45:07 PM | DELETE_COMPLETE      | AWS::IAM::Role        | function /ServiceRole (functionServiceRoleD810D9B4) 
 9/5 | 4:45:08 PM | ROLLBACK_COMPLETE    | AWS::CloudFormation::Stack | lambdaStack 
The stack named lambdaStack failed creation, it may need to be manually deleted from the AWS console: ROLLBACK_COMPLETE

Environment

• CLI Version : cdk --version = 1.13.1 (build 96cfc63), cat pacakge.json = all 1.13.1
• Framework Version: ?
• OS : Amazon Linux 2 (internal)
• Language : typescript

[eladb]

@ArendAMZN is this still an issue?

[ArendAMZN]

@iliapolo This is still an issue

I pushed this to the side to focus on other tasks (even though I still need this to work eventually), but I hit it again with today's task

 13/34 | 5:22:20 PM | CREATE_FAILED        | AWS::Lambda::Function   | CommentingFunction (CommentingFunction5C76628F) Your access has been denied by S3, please make sure your reque
st credentials have permission to GetObject for keacodebuildprevents-source-code-us-west-2-326471334405/dev. S3 Error Code: AccessDenied. S3 Error Message: Access Denied (Service: A
WSLambdaInternal; Status Code: 403; Error Code: AccessDeniedException; Request ID: 9fc6a3c6-7832-4186-94a0-562957ba0cbd)

[ArendAMZN]

I'm wondering if lambdaFunction.role is actually the thing that needs access to the bucket? Since it's accessing the bucket to create the lambda (rather than execute it) maybe it's using a different role?

[iliapolo]

Acking this issue.

@ArendAMZN Apologies for the long delay, will get to it this week promptly.

[iliapolo]

Hi @ArendAMZN

Im trying to figure out the correct way to configure this. Though honestly, i'm not sure this is even possible. Lambda requires the source code bucket to be in the same region as the lambda, it doesn't mention anything about requiring the same account, but that doesn't seem like a stretch to assume.

While I investigate this further, I do have a few observations.

The following code you posted:

var funcCrossPolicy = new iam.PolicyStatement()
funcCrossPolicy.addResources(s3Role.roleArn);
funcCrossPolicy.addActions('sts:AssumeRole');
lambdaFunction.addToRolePolicy(funcCrossPolicy);

Doesn't help because the lambda isn't actively attempting to assume any role, so giving it permissions to do so is void.

The entity that needs access to the bucket in the different account is actually the lambda service itself, not any specific function. I would try using iam.ServicePrincipal to grant the lambda.amazonaws.com service the required permissions.

Unfortunately I don't have a full solution for you just yet, but i wanted to give my input so maybe we can figure this out together :)

Also, any chance you elaborate on the use-case for attempting this? Doing cross-account references might not be the best road to follow. Perhaps we can think of a better way to achieve what you need.

[ArendAMZN]

Thank you for looking into it, I'll try using the service principal to see if that fixes things

Our use case is very similar to the one detailed here, https://aws.amazon.com/blogs/devops/aws-building-a-secure-cross-account-continuous-delivery-pipeline/

We have a tooling account as well as accounts for beta/gamma (or at least that's the final goal). The example deploys cloudformation cross account, and then builds the lambdas within the account. For our use case we'd to pass lambda references within cdk, so waiting for cloud formation to generate the lambdas doesn't work for us

Instead we have a code pipeline which works as follows (codecommit -> s3_bucket/beta -> lambda_beta -> s3_bucket/prod -> lambda_prod). The lambdas are in the corresponding beta/prod accounts but the code pipeline and s3_bucket are both in the tooling account.

One possible modification is I can have separate beta/prod buckets in the corresponding beta/prod accounts. That way I'm doing as cross account s3 deployment, rather than a cross account lambda build. Do you think that will be easier?

[ArendAMZN]

Here are the details of the 3 approaches I'm considering. Let me know which path you think should be investigated further. I can write more scripts recreating the issue if that would be helpful

1. User service principal

I updated the s3 role as follows, but I still get the same bug

const s3Role = new iam.Role(s3Stack, 'CrossAccountRole', {
    roleName: 's3CrossAccountRoleName',
    assumedBy: new iam.CompositePrincipal(
        new iam.ServicePrincipal('lambda.amazonaws.com'),
        new iam.AccountPrincipal(toolEnv['account']),
    )
});

2. Avoid cross account bucket retrieval

As noted earlier, I have a codepipeline (tooling account) and a lambda (beta account). Currently the s3 bucket is in the tooling account. So the pipeline deploys to the s3 bucket, and then the pipeline needs to make a cross account call for the lambda to update from the bucket (reaching back cross account to get the bucket). I can avoid some of this by moving the s3 bucket to the beta account. So the codepipeline needs to deploy cross account to s3, but the lambda doesn't need to pull from cross account to update the code

This was an improvement, cdk was able to deploy the codepipeline and lambda stacks, and the codepipeline was able to deploy new code updates cross account to the s3 bucket, but the codepipeline can't the lambda to update it's code from s3 (so the lambda is being triggered cross account, but the operation it needs to do is in the account)

Here's the error I get in codebuild when it tries to make the lambda refresh it's code

An error occurred (AccessDeniedException) when calling the UpdateFunctionCode operation: User: arn:aws:sts::776362166164:assumed-role/QueryServicePipeline-DeployLambdaRoleFA441CFA-WIL0IOV8TVKO/AWSCodeBuild-c955c070-0d9e-40d2-973f-6f6ddd5f692b is not authorized to perform: lambda:UpdateFunctionCode on resource: arn:aws:lambda:us-west-2:721320610352:function:queryServiceFunction-beta

Here's the piece of that role which I thought would give it permission

{
    "Action": "sts:AssumeRole",
    "Resource": "arn:aws:iam::721320610352:role/QueryServiceLambdaUpdateRole",
    "Effect": "Allow"
},

And the corresponding role in the beta account

{
    "Action": "lambda:updateFunctionCode",
    "Resource": "*",
    "Effect": "Allow"
}

I've previously followed a similar pattern pattern to get my pipeline to access repos cross account, but I'm not sure why it's different for lambda

{
    "Action": "sts:AssumeRole",
    "Resource": "arn:aws:iam::363575151959:role/sourcestacksourceactionrole44fd7f23045100c18826",
    "Effect": "Allow"
},
....
{
    "Action": [
        "codecommit:GetBranch",
        "codecommit:GetCommit",
        "codecommit:UploadArchive",
        "codecommit:GetUploadArchiveStatus",
        "codecommit:CancelUploadArchive"
    ],
    "Resource": "arn:aws:codecommit:us-west-2:363575151959:KeaQueryService",
    "Effect": "Allow"
}

3. Move functionUpdate call into the same account as the lambda

Since the pipeline can transfer the code to s3 cross account. I can have a separate process trigger on the s3 bucket update and update the lambda (all within the beta account). This way the codepipeline doesn't have to do a cross account call to make the lambda update it's code.

I think this would work (although I haven't tested it yet), but I'm not a fan of the pattern. It feels a little excessive to manage an additional "lambda updater" lambda in addition to my service lambdas. Also it hides the status of the deployment from the pipeline (i.e. the pipeline can be green before the deployment is actually complete)

[ArendAMZN]

We figured it out using option #2

The iam role I was using for CodeBuild had assume role permissions to update the code, however I actually needed to perform the assume role operation for that to work (this is different from codepipeline where it automatically assumes the roles)

Adding this to my build script fixed it

 - temp_role=$(aws sts assume-role --role-arn <insert role arn here> --role-session-name "codebuild-session")
 - export AWS_ACCESS_KEY_ID=$(echo $temp_role | jq .Credentials.AccessKeyId | xargs)
 - export AWS_SECRET_ACCESS_KEY=$(echo $temp_role | jq .Credentials.SecretAccessKey | xargs)
 - export AWS_SESSION_TOKEN=$(echo $temp_role | jq .Credentials.SessionToken | xargs)

[iliapolo]

@ArendAMZN Awesome! Happy you got it working

Closing this issue