Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add exec_group_compatible_with attribute #24964

Draft
wants to merge 15 commits into
base: master
Choose a base branch
from

Conversation

fmeum
Copy link
Collaborator

@fmeum fmeum commented Jan 18, 2025

This change implements the Per-target execution platform constraints for exec groups proposal by adding the exec_group_compatible_with attribute. It differs from the proposal in its accepted from in two ways:

  • During the implementation of the proposal, it was discovered that the test exec group always inherits toolchain requirements and constraints from the default exec group. This behavior makes it impossible to apply additional constraints to the default exec group via exec_compatible_with without also applying them to the test exec group. This preexisting behavior has mostly undesirable consequences: In a multi- or cross-platform build, where exec constraints matter, tests actions and build actions are

Fixes #23202
Fixes #23802

RELNOTES: The new exec_group_compatible_with attribute on all rules accepts a dictionary mapping exec group names to lists of additional constraints to request from the exec group's execution platform.

RELNOTES[INC]: Constraints and toolchain requirements added to the default exec group, for example via the toolchains parameter of the rule function or the exec_compatible_with attribute on all rules, no longer apply to the test exec group, which contains the test action on test rules. Instead, use exec_group_compatible_with to apply constraints and/or define an explicit test exec group with toolchain requirements on test rules that require it.

@fmeum fmeum force-pushed the 23802-add-exec-group-compatible-with branch 2 times, most recently from 3df50ad to b54072a Compare January 19, 2025 10:32
fmeum added 2 commits January 20, 2025 21:32
This makes `exec_properties` such as `no-remote-exec` effective for the main test spawn, where it previously only affected post-processing spawns such as test XML generation.
@keith
Copy link
Member

keith commented Jan 22, 2025

on this reproducer: #23202 (comment) w/ this change (and the exec_properties one) I get this crash:

FATAL: bazel crashed due to an internal error. Printing stack trace:
java.lang.RuntimeException: Unrecoverable error while evaluating node '' (requested by nodes ':all, excludedSubdirs=[], filteringPolicy=com.google.devtools.build.lib.pkgcache.FilteringPolicies$FilterManual@7dab797e')
        at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:551)
        at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:435)
        at java.base/java.util.concurrent.ForkJoinTask$AdaptedRunnableAction.exec(ForkJoinTask.java:1577)
        at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:507)
        at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1489)
        at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:2071)
        at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:2033)
        at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:187)
Caused by: net.starlark.java.eval.Starlark$UncheckedEvalException: InvalidStarlarkValueException thrown during Starlark evaluation (an initializer)
        at <starlark>.cc_test(<builtin>:0)
        at <starlark>.<toplevel>(/tmp/repro/BUILD.bazel:30)
Caused by: net.starlark.java.eval.Starlark$InvalidStarlarkValueException: invalid Starlark value: class java.util.ArrayList
        at net.starlark.java.eval.Starlark.checkValid(Starlark.java:136)
        at net.starlark.java.eval.Dict.lambda$copyOf$0(Dict.java:454)
        at com.google.common.collect.SingletonImmutableBiMap.forEach(SingletonImmutableBiMap.java:70)
        at net.starlark.java.eval.Dict.copyOf(Dict.java:451)
        at net.starlark.java.eval.Starlark.fromJava(Starlark.java:217)
        at com.google.devtools.build.lib.packages.Attribute$StarlarkComputedDefaultTemplate.computeValue(Attribute.java:1471)
        at com.google.devtools.build.lib.packages.Attribute$StarlarkComputedDefaultTemplate$1.compute(Attribute.java:1425)
        at com.google.devtools.build.lib.packages.Attribute$ComputationStrategy.computeValuesForAllCombinations(Attribute.java:1228)
        at com.google.devtools.build.lib.packages.Attribute$StarlarkComputedDefaultTemplate.computePossibleValues(Attribute.java:1446)
        at com.google.devtools.build.lib.packages.RuleClass.populateDefaultRuleAttributeValues(RuleClass.java:2438)
        at com.google.devtools.build.lib.packages.RuleClass.populateRuleAttributeValues(RuleClass.java:2208)
        at com.google.devtools.build.lib.packages.RuleClass.createRule(RuleClass.java:2158)
        at com.google.devtools.build.lib.packages.RuleFactory.createRule(RuleFactory.java:106)
        at com.google.devtools.build.lib.packages.RuleFactory.createAndAddRule(RuleFactory.java:141)
        at com.google.devtools.build.lib.analysis.starlark.StarlarkRuleClassFunctions$StarlarkRuleFunction.call(StarlarkRuleClassFunctions.java:1754)
        at net.starlark.java.eval.StarlarkCallable.fastcall(StarlarkCallable.java:87)
        at net.starlark.java.eval.Starlark.fastcall(Starlark.java:804)
        at net.starlark.java.eval.Eval.evalFastcall(Eval.java:821)
        at net.starlark.java.eval.Eval.evalCall(Eval.java:678)
        at net.starlark.java.eval.Eval.eval(Eval.java:547)
        at net.starlark.java.eval.Eval.exec(Eval.java:288)
        at net.starlark.java.eval.Eval.execStatements(Eval.java:83)
        at net.starlark.java.eval.Eval.execFunctionBody(Eval.java:67)
        at net.starlark.java.eval.StarlarkFunction.callWithArguments(StarlarkFunction.java:225)
        at net.starlark.java.eval.StarlarkFunction.positionalOnlyCall(StarlarkFunction.java:209)
        at net.starlark.java.eval.Starlark.positionalOnlyCall(Starlark.java:872)
        at net.starlark.java.eval.Starlark.execFileProgram(Starlark.java:1195)
        at com.google.devtools.build.lib.packages.PackageFactory.executeBuildFileImpl(PackageFactory.java:416)
        at com.google.devtools.build.lib.packages.PackageFactory.executeBuildFile(PackageFactory.java:377)
        at com.google.devtools.build.lib.skyframe.PackageFunction.loadPackage(PackageFunction.java:1188)
        at com.google.devtools.build.lib.skyframe.PackageFunction.compute(PackageFunction.java:539)
        at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:471)
        at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:435)
        at java.base/java.util.concurrent.ForkJoinTask$AdaptedRunnableAction.exec(ForkJoinTask.java:1577)
        at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:507)
        at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1489)
        at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:2071)
        at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:2033)
        at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:187)

i looked into this a bit last week and I believe it's because this function

return Dict.copyOf(mutability, (Map<?, ?>) x);
needs some sort of "deep copy" in this case, but I couldn't figure that out so I can't say for sure

@fmeum fmeum force-pushed the 23802-add-exec-group-compatible-with branch from 8528676 to c943de3 Compare January 22, 2025 12:30
@keith
Copy link
Member

keith commented Jan 22, 2025

verified on the latest commits here it works with this repro case now!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants