[mono][tasks] Extract EmitWasmBundle into generic EmitBundle task and enable bundling in mono self-contained library #84191

mdh1418 · 2023-03-31T22:01:50Z

Contributes to #79377

In order for Mono's self-contained shared|static library mode to truly be self-contained (i.e. not relying on external resources such as compiled managed assemblies existing on disk), bundling the resources into the shared library (i.e. storing the data of necessary resources directly inside the shared library itself) is necessary. Wasm and Wasi already has a form of bundling implemented in #82250.

This PR looks to extract the bundling logic out of wasm specific files to allow other mobile platforms leverage the technology, and incorporate bundling into mono's library mode built through #81919 with runtime initialization through #82253 and #83050.

This PR does the following:

Extract Wasm's bundling task into a more general task
Incorporates bundling into Mono's self-contained library mode
Expand bundling to handle various types of resources (assemblies + associated pdb, satellite assemblies, other data resources [i.e. runtimeconfig.bin/timezones])
- Defines new MonoBundle*Resource structs all aligned off of MonoBundledResource to handle various types
- Utilizes a hash table to store pointers MonoBundled*Resources
- Bootstrap hash table into preexisting bundle resource retrieval paths
Modifies EmitBundle input parameters
- List of files to bundle with possible RegisteredFile metadata designating what the resource should be registered under in the hashtable.
- Registration function symbol to associate with logic that registers bundled resources via new mono_bundled_resources_add bundling api.
- Optional file path to generate logic to preallocate MonoBundled*Resources and define passed in registration function
- Directory to place generated files
Output generated bundled resource files from EmitBundle task
- Appends DataSymbol, DataLenSymbol, DataLenSymbolValue, RegisteredName, and DestinationFile metadata
Deprecates old bundling api in favor of new bundling api

Example Files

snippet of source file containing byte array data + lengths
source file to register preallocated resource structs, including typedefs and resource symbols

Testing

Mono's self-contained library mode without bundling had been tested in #83050
It was done by building an Android Functional Test with changes that

Removed built in runtime initialization logic from AndroidAppBuilder's monodroid.c
Enable Library mode through NativeLib=shared and ForceFullAOT=true (in addition to other tweaks to generate the app in library mode)
Add a custom managed side API that is called from native
Pass the path to relevant resources (i.e. managed assemblies, runtimeconfig.bin) on disk into the environment variable DOTNET_LIBRARY_ASSEMBLY_PATH
Validate that the mono runtime is initialized and that the managed side API has been properly invoked.

In order to validate bundling, DOTNET_LIBRARY_ASSEMBLY_PATH was not set, and BundlesAssemblies was passed to the LibraryBuilder. The app was able to bundle assemblies + runtimeconfig.bin, initialize runtimeconfig, auto initialize mono runtime, and invoke the managed side API properly.

ghost · 2023-03-31T22:02:04Z

Tagging subscribers to this area: @directhex
See info in area-owners.md if you want to be subscribed.

Issue Details

null

Author:	mdh1418
Assignees:	-
Labels:	`area-Infrastructure-mono`
Milestone:	-

…meconfig data

src/tasks/LibraryBuilder/Templates/autoinit.c

src/mono/mono/component/mini-wasm-debugger.c

src/mono/mono/metadata/mono-debug.c

mdh1418 · 2023-06-09T17:12:40Z

/azp run runtime-wasm

azure-pipelines · 2023-06-09T17:12:52Z

Azure Pipelines successfully started running 1 pipeline(s).

mdh1418 · 2023-06-09T18:13:38Z

Looking at other instances where there is MONO_WEBCIL_IN_WASM_EXTENSION to assist in assembly searching, there is

runtime/src/mono/mono/mini/monovm.c

Lines 135 to 163 in 79fda00

    
           #ifdef ENABLE_WEBCIL 
        
           			else { 
        
           				/* /path/foo.dll -> /path/foo.webcil */ 
        
           				size_t n = strlen (fullpath); 
        
           				if (n < strlen(".dll")) 
        
           					continue; 
        
           				n -= strlen(".dll"); 
        
           				char *fullpath2 = g_malloc (n + strlen(".webcil") + 1); 
        
           				g_strlcpy (fullpath2, fullpath, n + 1); 
        
           				g_strlcpy (fullpath2 + n, ".webcil", strlen(".webcil") + 1); 
        
           				if (g_file_test (fullpath2, G_FILE_TEST_IS_REGULAR)) { 
        
           					MonoImageOpenStatus status; 
        
           					result = mono_assembly_request_open (fullpath2, &req, &status); 
        
           				} 
        
           				g_free (fullpath2); 
        
           				if (result) 
        
           					break; 
        
           				char *fullpath3 = g_malloc (n + strlen(MONO_WEBCIL_IN_WASM_EXTENSION) + 1); 
        
           				g_strlcpy (fullpath3, fullpath, n + 1); 
        
           				g_strlcpy (fullpath3 + n, MONO_WEBCIL_IN_WASM_EXTENSION, strlen(MONO_WEBCIL_IN_WASM_EXTENSION) + 1); 
        
           				if (g_file_test (fullpath3, G_FILE_TEST_IS_REGULAR)) { 
        
           					MonoImageOpenStatus status; 
        
           					result = mono_assembly_request_open (fullpath3, &req, &status); 
        
           				} 
        
           				g_free (fullpath3); 
        
           				if (result) 
        
           					break; 
        
           			} 
        
           #endif

and

runtime/src/mono/mono/metadata/assembly.c

Lines 2741 to 2754 in 79fda00

    
           #ifdef ENABLE_WEBCIL 
        
           	if (!corlib) { 
        
           		/* Maybe its in a bundle */ 
        
           		char *corlib_name = g_strdup_printf ("%s.webcil", MONO_ASSEMBLY_CORLIB_NAME); 
        
           		corlib = mono_assembly_request_open (corlib_name, &req, &status); 
        
           		g_free (corlib_name); 
        
           	} 
        
           	if (!corlib) { 
        
           		/* Maybe its in a bundle */ 
        
           		char *corlib_name = g_strdup_printf ("%s%s", MONO_ASSEMBLY_CORLIB_NAME, MONO_WEBCIL_IN_WASM_EXTENSION); 
        
           		corlib = mono_assembly_request_open (corlib_name, &req, &status); 
        
           		g_free (corlib_name); 
        
           	} 
        
           #endif

The new bundling apis rely on a hashtable, which has the feature of aliasing known assembly extensions .dll, .wasm, .webcil (if webcil is enabled) all to .dll, removing some confusion on what extension the bundled assembly resource has.

The latter specifically mentions bundle, so it seems "safe" to remove that block altogether, but not sure if the former is specific to bundling. It also relies on mono_assembly_request_open, and that has fallbacks if not found in a bundle, @lambdageek do you know if that is also bundle specific path? Should we be able to remove both blocks?

lateralusX

LGTM! Great work!

pavelsavara

just couple of lazy questions, looks great!

I will follow-up with #87284

src/mono/mono/component/mini-wasm-debugger.c

pavelsavara · 2023-06-12T08:20:41Z

src/mono/mono/metadata/mono-debug.c

-	if (!strcmp (bsymfile->aname, assembly_name))
-		return TRUE;
-#ifdef ENABLE_WEBCIL
-	const char *p = strstr (assembly_name, ".webcil");


Does (unbundled) webcil still work ?

Which tests would validate unbundled webcil scenario? I think I was having trouble knowing how to enable webcil when running wasm/wasi tests.

This function, however, is only invoked in the two bundling specific APIs right below.
The new bundling logic will alias all known extensions of assemblies .dll, .wasm, .webcil (if enabled) to .dll, so that helps eliminate a chunk of logic that had searched for the different variations, hence the logic to check for .webcil or .wasm is no longer needed in bundling search paths

webcil is currently useful for browser. The debugger probably doesn't know about the format difference and possibly not even about name difference. I don't know if we have automated tests for debugger on webcil.

mdh1418 · 2023-06-12T19:03:42Z

System.Runtime.Loader.Tests on wasm tests the satellite assembly bundling logic, registration and lookup are working as intended and all of those tests pass.

radical

review in progress..

radical · 2023-06-13T00:13:43Z

src/tasks/MonoTargetsTasks/EmitBundleTask/EmitBundleObjectFiles.cs

-namespace Microsoft.WebAssembly.Build.Tasks;
-
-public class EmitWasmBundleObjectFiles : EmitWasmBundleBase
+public class EmitBundleObjectFiles : EmitBundleBase


Use some namespace for this. can be done in a follow up PR, along with adding common namespace for the other tasks in this assembly.

src/mono/wasi/build/WasiApp.Native.targets

radical · 2023-06-13T00:22:59Z

src/tasks/MonoTargetsTasks/EmitBundleTask/EmitBundleBase.cs

+    private Dictionary<string, string> resourceDataSymbolDictionary = new();
+
+    private Dictionary<string, string[]> resourcesForDataSymbolDictionary= new();


Private instance fields should start with _ like _resourceDataSymbolDictionary.
And these two should be readonly.

Prefixed with _
What is the rationale for making them readonly? These variables are being modified within the Execute() method, is that considered the constructor for tasks, in which readonly fields should only be modified within as suggested here.

radical · 2023-06-13T00:30:08Z

src/tasks/MonoTargetsTasks/EmitBundleTask/EmitBundleBase.cs

+    /// Could have RegisteredName, otherwise it would be the filename.
+    /// RegisteredName should be prefixed with namespace in form of unix like path. For example: "/usr/share/zoneinfo/"
+    [Required]
+    public ITaskItem[] FilesToBundle { get; set; } = default!;


Suggested change

public ITaskItem[] FilesToBundle { get; set; } = default!;

public ITaskItem[] FilesToBundle { get; set; } = Array.Empty<ITaskItem>();

Changed, out of curiosity, why is this preferred over the default keyword?

src/tasks/MonoTargetsTasks/EmitBundleTask/EmitBundleBase.cs

radical · 2023-06-13T02:22:26Z

src/tasks/MonoTargetsTasks/EmitBundleTask/EmitBundleBase.cs

+                    string? managedAssemblyCulture = null;
+
+                    var resourceMetadataReader = PEReaderExtensions.GetMetadataReader(resourcePEReader);
+                    if (resourceMetadataReader.IsAssembly)


Does this return true for satellite assemblies also?

Yes, I believe it does. Put in some logging and saw that this block was hit for satellite assemblies as well

radical

review in progress..

src/tasks/MonoTargetsTasks/EmitBundleTask/EmitBundleObjectFiles.cs

radical · 2023-06-13T07:08:26Z

src/tasks/MonoTargetsTasks/EmitBundleTask/EmitBundleBase.cs

+                bundledResource.SetMetadata("RegisteredName", registeredName);
+            }
+
+            string resourceDataSymbol = $"bundled_resource_{ToSafeSymbolName(TruncateEncodedHash(Utils.ComputeHashEx(resourcePath, Utils.HashAlgorithmType.SHA256, Utils.HashEncodingType.Base64Safe), MaxEncodedHashLength))}";


The output filename should be more descriptive though, like:

string destinationFileName = $"{Path.GetFileName(resourcePath)}_{ToSafeSymbolName(TruncateEncodedHash(Utils.ComputeHashEx(resourcePath, Utils.HashAlgorithmType.SHA256, Utils.HashEncodingType.Base64Safe), MaxEncodedHashLength))}_bundle{GetOutputFileExtension()}";

EDIT: .. which would be useful when debugging a build. It might be even be useful to emit these into a subdir bundles.

The problem here is that we will share identical bundled assets so we just generate one bundled item instead of duplicates for identical content, that means multiple different resources will map the same destination file, so that is why we can't use the file name as part of the destination file name, only the hash. Inside the generated file (if its a source file) there is a list of all resources that maps the same destination file.

It might be useful to add more logging around this to aid potential debugging mapping input asset -> destination file, but we would need to keep the naming based on the unique content of the input asset in order to only emit one bundled resource for identical input assets. One example where we could have identical input is time zone info, in that case we will register the different time zones using its unique resource name in runtime, but we share one binary representation if several time zones end up with identical time zone data.

_resourcesForDataSymbolDictionary tracks which resource registered names have the same file contents and therefore have the same symbol names. I'll add logging around _resourcesForDataSymbolDictionary logic to note when multiple resources have been mapped to the same destination file.

maraf

Looks great!

mdh1418 · 2023-06-13T19:17:54Z

/azp run runtime-wasm

azure-pipelines · 2023-06-13T19:18:04Z

Azure Pipelines successfully started running 1 pipeline(s).

mdh1418 · 2023-06-20T19:42:35Z

Related failures:
runtime-wasm: #87423 known
runtime: #87505 fixed
Timeouts: #76454

mdh1418 added 3 commits March 31, 2023 16:32

[tasks] Move EmitWasmBundle out of wasm specific task

1d4e246

[tasks] Make EmitBundle more generic

2955cdb

[tasks] Generate EmitBundle task and replace invocations of old task

2d25efc

dotnet-issue-labeler bot added the area-Infrastructure-mono label Mar 31, 2023

ghost assigned mdh1418 Mar 31, 2023

This was referenced Apr 1, 2023

WasmTestOnBrowser-System.* test failures in CI #83655

Closed

Wasm debugger test timing out #83847

Closed

mdh1418 force-pushed the mono_library_mode_bundling branch from 92c21a6 to 6cf49fe Compare April 3, 2023 21:21

mdh1418 changed the title ~~Mono library mode bundling~~ [mono][tasks] Extract EmitWasmBundle into generic EmitBundle task and enable bundling in mono self-contained library Apr 3, 2023

mdh1418 added 5 commits April 3, 2023 17:30

Extend preloading assemblies with mono_assembly_open

d8d97e8

[LibraryBuilder] Integrate bundling

eb5e396

Refactor LibraryBuilder targets to isolate Bundling

a34c284

[mono] Add mono_get_bundled_assembly in preparation for bundled runti…

c1924b2

…meconfig data

[mono][librarymode] Bundle runtimeconfig.bin

9084234

mdh1418 force-pushed the mono_library_mode_bundling branch from 6cf49fe to 9084234 Compare April 3, 2023 21:32

mdh1418 marked this pull request as ready for review April 3, 2023 22:03

mdh1418 requested review from radical, lewing, pavelsavara, kg, vargaz, lambdageek, SamMonoRT and marek-safar as code owners April 3, 2023 22:03

mdh1418 requested review from maraf and lateralusX April 3, 2023 22:03

mdh1418 mentioned this pull request Apr 3, 2023

[mono] Add support for library mode in the aot compiler #79377

Closed

9 tasks

build-analysis bot mentioned this pull request Apr 4, 2023

IOException running NuGet-Migrations during tests in dotnet CLI first run #80619

Closed

lateralusX reviewed Apr 4, 2023

View reviewed changes

src/tasks/LibraryBuilder/Templates/autoinit.c Outdated Show resolved Hide resolved

This was referenced Jun 9, 2023

MsQuicPlatformDetectionTests failures in CI #87275

Closed

Work item tests crash on OSX x64 Debug #87287

Closed

lateralusX reviewed Jun 9, 2023

View reviewed changes

src/mono/mono/component/mini-wasm-debugger.c Outdated Show resolved Hide resolved

lateralusX reviewed Jun 9, 2023

View reviewed changes

src/mono/mono/metadata/mono-debug.c Outdated Show resolved Hide resolved

lateralusX reviewed Jun 9, 2023

View reviewed changes

src/mono/mono/metadata/mono-debug.c Outdated Show resolved Hide resolved

mdh1418 mentioned this pull request Jun 9, 2023

[browser][debugger][MT] CallFunctionOnTests.RunOnInvalidCfoId #87315

Closed

build-analysis bot mentioned this pull request Jun 9, 2023

Unable to clean up running docker containers #87323

Closed

Split MonoBundledAssemblyResource getter

c2c69ab

lateralusX approved these changes Jun 12, 2023

View reviewed changes

pavelsavara approved these changes Jun 12, 2023

View reviewed changes

radical reviewed Jun 13, 2023

View reviewed changes

src/tasks/MonoTargetsTasks/EmitBundleTask/EmitBundleObjectFiles.cs Show resolved Hide resolved

radical reviewed Jun 13, 2023

View reviewed changes

maraf approved these changes Jun 13, 2023

View reviewed changes

mdh1418 added 2 commits June 13, 2023 14:47

Addressing more feedback

45d2f8b

Validate OutputDirectory

b377ff4

mdh1418 merged commit 615eb4a into dotnet:main Jun 20, 2023
176 of 191 checks passed

mdh1418 deleted the mono_library_mode_bundling branch June 20, 2023 19:42

dotnet locked as resolved and limited conversation to collaborators Jul 21, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[mono][tasks] Extract EmitWasmBundle into generic EmitBundle task and enable bundling in mono self-contained library #84191

[mono][tasks] Extract EmitWasmBundle into generic EmitBundle task and enable bundling in mono self-contained library #84191

mdh1418 commented Mar 31, 2023 •

edited

ghost commented Mar 31, 2023

mdh1418 commented Jun 9, 2023

azure-pipelines bot commented Jun 9, 2023

mdh1418 commented Jun 9, 2023 •

edited

lateralusX left a comment

pavelsavara left a comment

pavelsavara Jun 12, 2023

mdh1418 Jun 12, 2023 •

edited

pavelsavara Jun 12, 2023

mdh1418 commented Jun 12, 2023

radical left a comment

radical Jun 13, 2023

radical Jun 13, 2023

mdh1418 Jun 13, 2023

radical Jun 13, 2023

mdh1418 Jun 13, 2023

radical Jun 13, 2023

mdh1418 Jun 13, 2023

radical left a comment

radical Jun 13, 2023 •

edited

lateralusX Jun 13, 2023 •

edited

lateralusX Jun 13, 2023 •

edited

mdh1418 Jun 13, 2023

maraf left a comment

mdh1418 commented Jun 13, 2023

azure-pipelines bot commented Jun 13, 2023

mdh1418 commented Jun 20, 2023

		private Dictionary<string, string> resourceDataSymbolDictionary = new();

		private Dictionary<string, string[]> resourcesForDataSymbolDictionary= new();

	public ITaskItem[] FilesToBundle { get; set; } = default!;
	public ITaskItem[] FilesToBundle { get; set; } = Array.Empty<ITaskItem>();

[mono][tasks] Extract EmitWasmBundle into generic EmitBundle task and enable bundling in mono self-contained library #84191

[mono][tasks] Extract EmitWasmBundle into generic EmitBundle task and enable bundling in mono self-contained library #84191

Conversation

mdh1418 commented Mar 31, 2023 • edited

Example Files

Testing

ghost commented Mar 31, 2023

mdh1418 commented Jun 9, 2023

azure-pipelines bot commented Jun 9, 2023

mdh1418 commented Jun 9, 2023 • edited

lateralusX left a comment

Choose a reason for hiding this comment

pavelsavara left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mdh1418 Jun 12, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mdh1418 commented Jun 12, 2023

radical left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

radical left a comment

Choose a reason for hiding this comment

radical Jun 13, 2023 • edited

Choose a reason for hiding this comment

lateralusX Jun 13, 2023 • edited

Choose a reason for hiding this comment

lateralusX Jun 13, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

maraf left a comment

Choose a reason for hiding this comment

mdh1418 commented Jun 13, 2023

azure-pipelines bot commented Jun 13, 2023

mdh1418 commented Jun 20, 2023

mdh1418 commented Mar 31, 2023 •

edited

mdh1418 commented Jun 9, 2023 •

edited

mdh1418 Jun 12, 2023 •

edited

radical Jun 13, 2023 •

edited

lateralusX Jun 13, 2023 •

edited

lateralusX Jun 13, 2023 •

edited