Metadata blob is too large, exceeding 16MB

[TickThunder]

Description

When I try to build a NativeAOT program, ILC reports that the metadata blob exceeds 16MB. Are there any known solutions for this?

EXEC : error : Metadata blob exceeded the addressing range (allowed: 16777215, actual: 25769597)
System.InvalidOperationException: Metadata blob exceeded the addressing range (allowed: 16777215, actual: 25769597)
at ILCompiler.GeneratingMetadataManager.ComputeMetadata[TPolicy](TPolicy, NodeFactory, Byte[]&, List1&, List1&, List1&, List1&) + 0x103b
at ILCompiler.AnalysisBasedMetadataManager.ComputeMetadata(NodeFactory, Byte[]&, List1&, List1&, List1&, List1&) + 0x55
at ILCompiler.DependencyAnalysis.MetadataNode.GetData(NodeFactory, Boolean) + 0xca
at ILCompiler.DependencyAnalysis.ObjectWriter.EmitObject(String, IReadOnlyCollection`1, NodeFactory, ObjectWritingOptions, IObjectDumper, Logger) + 0x50f
at ILCompiler.RyuJitCompilation.CompileInternal(String, ObjectDumper) + 0xad
at ILCompiler.Compilation.ILCompiler.ICompilation.Compile(String, ObjectDumper) + 0x2d
at ILCompiler.Program.Run() + 0x2772
at ILCompiler.ILCompilerRootCommand.<>c__DisplayClass227_0.<.ctor>b__0(ParseResult result) + 0x315

Related issue

@MichalStrehovsky

ConsoleApp10.ilc.rsp.txt

Reproduction Steps

use command 'dotnet publish'

Expected behavior

Compilation successful.

Actual behavior

Compilation failed.

Regression?

No response

Known Workarounds

No response

Configuration

No response

Other information

No response

[MichalStrehovsky]

Can you paste the contents of you *.ilc.rsp file? It should in the obj directory.

[TickThunder]

Can you paste the contents of you *.ilc.rsp file? It should in the obj directory.

ConsoleApp10.ilc.rsp.txt

[MichalStrehovsky]

There's one suspicious thing in this and that's the instruction to treat Assembly-CSharp as a root (probably with TrimmerRootAssembly in the project). Is the assembly big? TrimmerRootAssembly is not very recommended (in general, untrimmable code that needs TrimmerRootAssembly is not recommended).

One more diagnostic file that would help troubleshooting this is the metadata log. It can be generated by setting the IlcGenerateMetadataLog property to true. This should produce a ConsoleApp10.metadata.csv file under the obj directory. It lists everything that is part of the 25 MB blob. We could see what thing is generating so much metadata.

[TickThunder]

There's one suspicious thing in this and that's the instruction to treat Assembly-CSharp as a root (probably with TrimmerRootAssembly in the project). Is the assembly big? TrimmerRootAssembly is not very recommended (in general, untrimmable code that needs TrimmerRootAssembly is not recommended).

One more diagnostic file that would help troubleshooting this is the metadata log. It can be generated by setting the IlcGenerateMetadataLog property to true. This should produce a ConsoleApp10.metadata.csv file under the obj directory. It lists everything that is part of the 25 MB blob. We could see what thing is generating so much metadata.

I am unable to provide the metadata log due to NDA. I have no way to determine which functions and types in Assembly-CSharp need to be used, so it is correct to set TrimmerRootAssembly. Our project's Assembly-CSharp is indeed very large.

[MichalStrehovsky]

I am unable to provide the metadata log due to NDA

The file is a CSV. The first column has numbers that will look like this:

38000010
38000106
380001f8

They are in hex. The first two digits are handle types, ignore them. The subsequent 6 digits are offsets within the blob. Size can be computed by subtracting the offset from previous offset (so the size of the first one in my example is 0x10 bytes, second is 0x106-0x10=0xF6 bytes, etc.). Look what are the big ones. Are there e.g. very long strings (they would map to const string in C#)?

If it's just "there's too much little stuff", you really need to allow the compiler to trim stuff. We don't support native AOT without trimming (setting PublishTrimmed to false errors out the compilation). But there are definitely other ways how the compiler can be put into a position of not being able to trim stuff that is similar to PublishTrimmed set to false. Rooting massive assemblies is one of the ways (how big is the assembly?).

[TickThunder]

I am unable to provide the metadata log due to NDA

The file is a CSV. The first column has numbers that will look like this:

38000010
38000106
380001f8

They are in hex. The first two digits are handle types, ignore them. The subsequent 6 digits are offsets within the blob. Size can be computed by subtracting the offset from previous offset (so the size of the first one in my example is 0x10 bytes, second is 0x106-0x10=0xF6 bytes, etc.). Look what are the big ones. Are there e.g. very long strings (they would map to const string in C#)?

If it's just "there's too much little stuff", you really need to allow the compiler to trim stuff. We don't support native AOT without trimming (setting PublishTrimmed to false errors out the compilation). But there are definitely other ways how the compiler can be put into a position of not being able to trim stuff that is similar to PublishTrimmed set to false. Rooting massive assemblies is one of the ways (how big is the assembly?).

Our Assembly-CSharp.dll is around 60MB in size. I removed the name from the uploaded metadata log.

Our xxx.csproj like this

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Library</OutputType>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>
    <PublishAot>True</PublishAot>
  </PropertyGroup>

  <ItemGroup>
    <Reference Include="Assembly-CSharp">
      <HintPath>Q:\Build\StandaloneWindows64\16\Windows\App\TempScripts\Managed\Assembly-CSharp.dll</HintPath>
    </Reference>
  </ItemGroup>

  <ItemGroup>
    <TrimmerRootAssembly Include="Assembly-CSharp" />
  </ItemGroup>

</Project>

metadatalog.csv.zip

[MichalStrehovsky]

There's nothing out of the ordinary in the metadata log (nothing extremely large, just lots of small things).

Your only option is to drop the TrimmerRootAssembly and be more targeted in what needs to be rooted. The compiler doesn't perform well if everything within the assembly needs to be considered target of reflection. For example, small methods that would normally be just inlined at their callsites need to be generated as standalone method bodies (in addition to being inlined everywhere). The compiler cannot assume anything about values of fields. Etc.

Things get ridiculously large. There's a big difference between "calling a method/accessing a field in code" and "calling a method/accessing a field using reflection". TrimmerRootAssembly puts everything on the later plan. 60 MB of IL that needs to be considered not just used, but used by reflection is too much for the compiler to handle.

I'm going to close this since we have dotnet/runtimelab#1581 tracking maybe solving this (although the proposed solution there is no longer viable since we ran out of bits). That issue was also filed in response to someone who ran into this by putting TrimmerRootAssembly on entire ASP.NET MVC, ML.NET, Roslyn, and I'm sure I'm forgetting some. It simply runs into implementation limits.

[TickThunder]

I'm going to close this since we have dotnet/runtimelab#1581 tracking maybe solving this (although the proposed solution there is no longer viable since we ran out of bits).

The maximum value of HandleType is 0x3F (0011 1111), which theoretically means there should still be one bit available for the offset. Why do you say the bits are already exhausted?

[MichalStrehovsky]

I'm going to close this since we have dotnet/runtimelab#1581 tracking maybe solving this (although the proposed solution there is no longer viable since we ran out of bits).

The maximum value of HandleType is 0x3F (0011 1111), which theoretically means there should still be one bit available for the offset. Why do you say the bits are already exhausted?

The next bit is going to be taken whenever we need to do schema update and uncomment this (there's an open issue asking to expose MethodImpl in reflection and when that happens, we need to start generating this):

runtime/src/coreclr/tools/Common/Internal/Metadata/NativeFormat/Generator/SchemaDef.cs

Lines 651 to 657 in a593d9c

	/* COMPLETENESS new RecordDef(
	name: "MethodImpl",
	members: new MemberDef[] {
	new MemberDef("MethodBody", MethodDefOrRef, MemberDefFlags.RecordRef),
	new MemberDef("MethodDeclaration", MethodDefOrRef, MemberDefFlags.RecordRef),
	}
	),*/