[AVM CI Environment Issue]: Deployment step is failing when template size exceeds the maximum size of 4 MB

[kpantos]

Check for previous/existing GitHub issues

• I have checked for previous/existing GitHub issues

Issue Type?

Bug

Description

The deployment part of the CI workflow fails at the Validate Template File step when template size exceeds the maximum size of 4 MB.

WARNING: The request content size exceeds the maximum size of 4 MB.
  Write-Error: /home/runner/work/_temp/36d402fe-2384-47c8-90c9-250096c1bdbc.ps1:70
  Line |
    70 |  Test-TemplateDeployment @functionInput -Verbose
       |  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
       | Template is not valid.
  Error: Error: The process '/usr/bin/pwsh' failed with exit code 1

job-logs.txt

Manual deployment using Bicep build -f main.test.bicep and az deployment sub create works and deploys the test correctly.

[kpantos]

@Azure/avm-core-team-technical-bicep any updates on this issue?

[jtracey93]

Hey @k pantos, can you share a link to the fork/branch where you are working on and what file this is failing for please?

@ReneHezser would you be able to investigate once we get the details shared above?

[kpantos]

Hey @jtracey93, you can see the issue here https://github.com/kpantos/bicep-registry-modules/actions/runs/12521716173/job/34929149890

[ReneHezser]

I see the compiled tests are 3927 kB in size (which is close to the maximum ARM templates support). Let me check the pipeline.

[kpantos]

That's what I thought initially, but running the test manually through command line succeeds.

[ReneHezser]

I've been looking at the CI and see that the test-deployment returns with the error about the 4MB limit.

Write-Information "Starting test-deployment"
$DeploymentInputs = @{
    "TemplateFile"="C:\Code\kpantos\bicep-registry-modules\avm\ptn\aca-lza\hosting-environment\tests\e2e\waf-aligned\main.test.bicep";
    "DeploymentName"="a-p-al-he-waf-aligned-20250107T1601239878Z";
    "OutVariable"="ValidationErrors";
    "Verbose"=$true;
}
$DeploymentMetadataLocation="germanywestcentral"
# uses https://learn.microsoft.com/en-us/powershell/module/az.resources/test-azdeployment?view=azps-13.0.0
$res = Test-AzSubscriptionDeployment @DeploymentInputs -Location $DeploymentMetadataLocation
if ($res.Message) { Write-Warning $res.Message }

Write-Information "Starting manual deployment"
az deployment sub create --name $DeploymentInputs.DeploymentName --location $DeploymentMetadataLocation --template-file $DeploymentInputs.TemplateFile --what-if

The first part is what is done in the pipeline (or when you test locally via Test-ModuleLocally) and returns with WARNING: The request content size exceeds the maximum size of 4 MB..

I don't see how we can get around this without removing the testing .
But I am open for suggestions :-)

[kpantos]

I'm wondering what is different from doing Bicep build -f main.test.bicep and az deployment sub create -f .\tests\e2e\defaults\main.test.bicep -l northeurope which is working and deploys the test correctly.

Is the CI/CD deployment pipeline leveraging my linked templates https://learn.microsoft.com/en-us/azure/azure-resource-manager/templates/linked-templates?tabs=azure-powershell? or is it trying to validate one single arm template generated by bicep buiild?

[jtracey93]

Is it all the tests or just the WAF aligned one as @ReneHezser showed in his testing?

Also has anyone tried/tested building this on a Linux device as that's what the runners will do? Wondering if something to do with case and line endings handling may be in play here.

Regardless getting that close to the 4mb limit is a recipe for future issues anyway. Say I the bicep PG add something to the language in an upcoming release that bloats templates even slightly, then this will cause the issue to reoccur.

Only long term solution in our control is to advise maybe splitting this into multiple pattern modules to shrink the size a bit

[kpantos]

It's all the tests that are failing.

I understand that the avm module needs to be <4MB as it needs to be compiled and stored into the registry (which is by the way a serious limitation for the program as it restricts the composing of more complex solutions). Even splitting into multiple pattern modules, it would defeat the LZA pattern's purpose if these can't be composed together to build a larger pattern but that's another discussion for another time since in this case the module is below that threshold.

The problem probably arises when the tests are also compiled into a single json file together with the module - which for testing it doesn't have to be so. Linked templates can be used for testing which would avoid going over the limit.

[jtracey93]

Hey @kpantos,

Thanks for clarifying 😃

Just to clarify something around the 4MB limit, this isn't an AVM only thing or a container registry thing for storing modules, it's a hard limit in ARM itself for any one template. And because bicep compiles everything into a single ARM template (JSON file), due to the way it currently is designed and engineered, the 4MB limit can be hit more often in bicep than it did in ARM. As you can see lots of long running discussion on the issue over on the Bicep repository for this (the correct place to vent about the 4MB limit 😁 ) Azure/bicep#5890

@ReneHezser lets get this one on the list for this weeks AVM Bicep core team call and see what we can do, if anything.

Finally, @kpantos, whilst the template today is just under the limit, what about in the future, being that close means that we could just be kicking the can down the road and the 4MB limit issue will come back for "real" next time you want to add something to the module. Is there any optimisations and reductions you can make to any of the files used to compose the module, e.g. trimming descriptions, removing whitespace, removing duplicated code etc.? (I actually had to do this for an ALZ module, outside of AVM, as we ran into the 4MB limit 😢 )

[kpantos]

Hey @jtracey93

I know the 4MB limit is not an AVM thing, and my intention was not to vent about it 😁 but to urge you to bring this up with the ARM team. Leveraging the importance of AVM program and its requirement to have a single deployment - so that it can be re-used, you can put more pressure on the team to increase this limit than me (in this case the ARM team can just argue the use of different deployments).

Also, there are no plans to add anything more to it (unless a new requirement like generating a demo https certificate on the fly arises again 😉). But the pattern's purpose is to provide for a viable, production ready, well architected Azure Container Apps hosting environment (aka LZA) so cutting more stuff from the template would defeat its purpose. The only real alternative would be to pull the pattern down and just use AVM resource modules for the LZA deployment.

Having said that I already went through an optimization cycle since yesterday and now 2 out of 3 tests are passing. The hub-spoke one though has a dependency in a hub deployment which contains a "heavy" Firewall avm resource module which even with its minimum configuration increases the template file by about 1MB. So, I have two options here, either delete the test all together or provide a way in the pipeline to deploy dependencies in a different deployment (something like a pre-deploy hook?) so that the dependencies template is not added to the pattern module under testing.

Thoughts?

[jtracey93]

Hey @kpantos,

No worries, although it did seem like a vit of a vent at AVM 😄

We have indeed spoken to the ARM/Bicep PG team for this and they are very, very aware of the 4MB limit and they themselves would love to change the limit but its a bit more complicated than a simple increase. Its something they are considering but it wont be anytime soon, so in the meantime we have to deal with it as a known constraint 👍

Good news on getting 2 out of the 3 tests working. Leave it with us for now, we are meeting as a team tomorrow and will discuss options there and then share back. We don't meet till the afternoon/evening as a JFYI so likely an update on Friday

Thanks for your patience and collaboration here