Python: CdkFunction.environment is dereferenced prematurely instead of returning IResolvable

[object-Object]

Describe the bug

CdkFunction.environment is a lazily evaluated value which can be resolved to CfnFunction.EnvironmentProperty.

In TypeScript and Go, environment returns LazyAny and typeregistry.anonymousProxy respectively, both of which can then be resolved with stack.resolve(environment).

In Python, environment returns an actual EnvironmentProperty instead of a reference, with variables=None. This resolves to an empty dict.

Expected Behavior

I expected CfnFunction.environment to be an IResolvable value which could then be resolved with stack.resolve(environment).

Current Behavior

CfnFunction.environment is an empty EnvironmentProperty value with no reference, so it's not possible to get the environment variables from this value.

JSII_DEBUG output from Go:

> {"api":"get","property":"environment","objref":{"$jsii.byref":"aws-cdk-lib.aws_lambda.CfnFunction@10046"}}
[@jsii/kernel] get { '$jsii.byref': 'aws-cdk-lib.aws_lambda.CfnFunction@10046' } environment
[@jsii/kernel] value: LazyAny {
  producer: { produce: [Function: produce] },
  cache: false,
  creationStack: [ 'Execute again with CDK_DEBUG=true to capture stack traces' ],
  options: {}
}
[@jsii/kernel] serialize LazyAny {
  producer: { produce: [Function: produce] },
  cache: false,
  creationStack: [ 'Execute again with CDK_DEBUG=true to capture stack traces' ],
  options: {}
} {
  serializationClass: 'Struct',
  typeRef: {
    type: { fqn: 'aws-cdk-lib.aws_lambda.CfnFunction.EnvironmentProperty' },
    optional: true
  }
} {
  serializationClass: 'RefType',
  typeRef: { type: { fqn: 'aws-cdk-lib.IResolvable' }, optional: true }
}
[@jsii/kernel] Returning value type by reference
[@jsii/kernel] ret: {
  '$jsii.byref': 'Object@10047',
  '$jsii.interfaces': [ 'aws-cdk-lib.aws_lambda.CfnFunction.EnvironmentProperty' ]
}
< {"ok":{"value":{"$jsii.byref":"Object@10047","$jsii.interfaces":["aws-cdk-lib.aws_lambda.CfnFunction.EnvironmentProperty"]}}}

From Python:

> {"objref":{"$jsii.byref":"aws-cdk-lib.aws_lambda.CfnFunction@10046"},"property":"environment","api":"get"}
[@jsii/kernel] get { '$jsii.byref': 'aws-cdk-lib.aws_lambda.CfnFunction@10046' } environment
[@jsii/kernel] value: LazyAny {
  producer: { produce: [Function: produce] },
  cache: false,
  creationStack: [ 'Execute again with CDK_DEBUG=true to capture stack traces' ],
  options: {}
}
[@jsii/kernel] serialize LazyAny {
  producer: { produce: [Function: produce] },
  cache: false,
  creationStack: [ 'Execute again with CDK_DEBUG=true to capture stack traces' ],
  options: {}
} {
  serializationClass: 'Struct',
  typeRef: {
    type: { fqn: 'aws-cdk-lib.aws_lambda.CfnFunction.EnvironmentProperty' },
    optional: true
  }
} {
  serializationClass: 'RefType',
  typeRef: { type: { fqn: 'aws-cdk-lib.IResolvable' }, optional: true }
}
[@jsii/kernel] Returning value type by reference
[@jsii/kernel] ret: {
  '$jsii.byref': 'Object@10047',
  '$jsii.interfaces': [ 'aws-cdk-lib.aws_lambda.CfnFunction.EnvironmentProperty' ]
}
< {"ok":{"value":{"$jsii.byref":"Object@10047","$jsii.interfaces":["aws-cdk-lib.aws_lambda.CfnFunction.EnvironmentProperty"]}}}
> {"objref":{"$jsii.byref":"Object@10047"},"property":"variables","api":"get"}
[@jsii/kernel] get { '$jsii.byref': 'Object@10047' } variables
[@jsii/kernel] value: undefined
[@jsii/kernel] serialize undefined {
  serializationClass: 'Map',
  typeRef: { type: { collection: [Object] }, optional: true }
} {
  serializationClass: 'RefType',
  typeRef: { type: { fqn: 'aws-cdk-lib.IResolvable' }, optional: true }
}
[@jsii/kernel] ret: undefined
< {"ok":{}}

Note the extra request at the end of the Python logs. This is from the exact same line of code, cfnFunction.Environment() and cfn_function.environment respectively.

Reproduction Steps

Paste each block into the stack file in the project created by cdk init for that language, then run cdk ls.

Python:

from aws_cdk import Stack, aws_lambda as lambda_
from constructs import Construct

class CdkPyStack(Stack):
    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        func = lambda_.Function(
            self,
            "Function",
            code=lambda_.Code.from_inline("_"),
            runtime=lambda_.Runtime.PYTHON_3_10,
            handler="_",
            environment={"KEY": "value"},
        )

        cfn_function = func.node.default_child
        assert isinstance(cfn_function, lambda_.CfnFunction)
        environment = cfn_function.environment
        print(environment)
        print(self.resolve(environment))

Output:

EnvironmentProperty()
{}
CdkPyStack

TypeScript (working, for comparison):

import * as cdk from 'aws-cdk-lib';
import { Construct } from 'constructs';
import * as lambda from 'aws-cdk-lib/aws-lambda';

export class CdkTsStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    let func = new lambda.Function(this, "Function", {
      code: lambda.Code.fromInline("_"),
      runtime: lambda.Runtime.PYTHON_3_10,
      handler: "_",
      environment: { KEY: "value" },
    });

    let cfnFunction = func.node.defaultChild as lambda.CfnFunction;
    let environment = cfnFunction.environment;
    console.log(environment);
    console.log(this.resolve(environment));
  }
}

Output:

LazyAny {
  producer: { produce: [Function: produce] },
  cache: false,
  creationStack: [ 'Execute again with CDK_DEBUG=true to capture stack traces' ],
  options: {}
}
{ variables: { KEY: 'value' } }
CdkTsStack

Possible Solution

Here's what I've been able to figure out by stepping through with a debugger. Hopefully this helps.

CfnFunction.environment is a property which calls jsii.get(), aka kernel.get().

jsii/packages/@jsii/python-runtime/src/jsii/_kernel/__init__.py

Lines 358 to 366 in 14b5ed2

	@_dereferenced
	def get(self, obj: Any, property: str) -> Any:
	response = self.provider.get(
	GetRequest(objref=obj.__jsii_ref__, property=property)
	)
	if isinstance(response, Callback):
	return _callback_till_result(self, response, GetResponse)
	else:
	return response.value

The value returned from kernel.get() is an ObjRef, which is passed to _reference_map.resolve_reference() via the @_dereferenced decorator and _recursize_dereference().

jsii/packages/@jsii/python-runtime/src/jsii/_kernel/__init__.py

Lines 133 to 151 in 14b5ed2

	def _recursize_dereference(kernel: "Kernel", d: Any) -> Any:
	if isinstance(d, dict):
	return {k: _recursize_dereference(kernel, v) for k, v in d.items()}
	elif isinstance(d, list):
	return [_recursize_dereference(kernel, i) for i in d]
	elif isinstance(d, ObjRef):
	return _reference_map.resolve_reference(kernel, d)
	elif isinstance(d, EnumRef):
	return _recursize_dereference(kernel, d.ref)(d.member)
	else:
	return d
	def _dereferenced(fn: Callable) -> Callable:
	@functools.wraps(fn)
	def wrapped(kernel: "Kernel", *args: Any, **kwargs: Any):
	return _recursize_dereference(kernel, fn(kernel, *args, **kwargs))
	return wrapped

The ref is something like Object@10047, so class_fqn is Object. The condition on line 104-106 below evaluates to True and it ends up creating and returning an EnvironmentProperty value with no data (because it hasn't been resolved with CDK yet).

jsii/packages/@jsii/python-runtime/src/jsii/_reference_map.py

Lines 102 to 128 in 14b5ed2

	elif class_fqn == "Object":
	# If any one interface is a struct, all of them are guaranteed to be (Kernel invariant)
	if ref.interfaces is not None and any(
	fqn in _data_types for fqn in ref.interfaces
	):
	# Ugly delayed import here because I can't solve the cyclic
	# package dependency right now :(.
	from ._runtime import python_jsii_mapping
	structs = [_data_types[fqn] for fqn in ref.interfaces]
	remote_struct = _FakeReference(ref)
	if len(structs) == 1:
	struct = structs[0]
	else:
	struct = new_combined_struct(structs)
	return struct(
	**{
	python_name: kernel.get(remote_struct, jsii_name)
	for python_name, jsii_name in (
	python_jsii_mapping(struct) or {}
	).items()
	}
	)
	else:
	return InterfaceDynamicProxy(self.build_interface_proxies_for_ref(ref))

Additional Information/Context

My use case is to create a custom aspect which adds default environment variables to all Lambdas in a stack. The aspect shouldn't overwrite variables set directly on a function, so it needs to know which ones already exist, if any.

SDK version used

JSII 1.84.0, CDK lib and cli 2.84.0, Constructs 10.2.55

Environment details (OS name and version, etc.)

Windows 10.0.14393 Build 14393, Python 3.11.3

[object-Object]

Update: it's possible to work around this issue by calling internal JSII functions with the below code, but it feels fragile.

This works by essentially intercepting the output of jsii.get() before it goes through _recursize_dereference(), changing the type of the reference to IResolvable, and then dereferencing it.

from jsii._kernel.types import GetRequest, ObjRef
from jsii._reference_map import InterfaceDynamicProxy, resolve_reference
from aws_cdk.aws_lambda import CdkFunction

obj: CdkFunction = ...

response = jsii.kernel.provider.get(
    GetRequest(objref=obj.__jsii_ref__, property="environment")
).value

return resolve_reference(jsii.kernel, ObjRef(response.ref, ["aws-cdk-lib.IResolvable"]))