Fails for Windows ARM64 GitHub runners

[jxy-s]

According to the following job it appears that setup-bazel is not functioning when using Windows ARM64 GitHub runners:
https://github.com/archonitelabs/radiant-cpp/actions/runs/9359175843/job/25800924517

See also:
https://github.blog/2024-06-03-arm64-on-github-actions-powering-faster-more-efficient-build-systems/

[jxy-s]

The workflow is using an older version, bazel-contrib/[email protected] - forgive me if this is already fixed. I can retest with the latest version in a bit.

[jxy-s]

Confirmed running with latest version shows the same issue: https://github.com/archonitelabs/radiant-cpp/actions/runs/9371722658/job/25801604485?pr=18

For posterity, the extra slash here is necessary: bazel test --test_output=all ///... - the non-ARM64 one works fine, and this is necessary to work around an idiosyncrasy with PowerShell on windows:

https://github.com/archonitelabs/radiant-cpp/blob/a9fa01fcd620944bb2052448797594c7697bb5bd/.github/workflows/unit-tests.yaml#L54-L83

[jxy-s]

Here is a run with debugging enabled: https://github.com/archonitelabs/radiant-cpp/actions/runs/9371722658/job/25801888908

[p0deje]

@jxy-s Your test fails with Error: bash: command not found - it doesn't seem to be related to setup-bazel.

[jxy-s]

🤔 I see, I guess I easily overlooked that since the other non-ARM windows machine worked with it. Let me try an alternative.

[jxy-s]

Okay so using cmd instead of bash shows: https://github.com/archonitelabs/radiant-cpp/actions/runs/9372646381/job/25804580277?pr=18

Run bazel test --test_output=all //...
'bazel' is not recognized as an internal or external command,
operable program or batch file.
Error: Process completed with exit code 1.

[p0deje]

That's because there is no Bazel installed on GitHub runner, even though it's pre-installed on Windows X64. If you want for the action to install one for you, you need to pass bazelisk-version: 1x input.

[jxy-s]

https://github.com/archonitelabs/radiant-cpp/actions/runs/9372804523/job/25805096299?pr=18

  Error: Error: Unable to find Bazelisk version 1.x for platform windows/arm64
      at downloadBazelisk (C:\a\_actions\bazel-contrib\setup-bazel\0.8.5\dist\index.js:90:1)
      at processTicksAndRejections (node:internal/process/task_queues:95:5)
      at setupBazelisk (C:\a\_actions\bazel-contrib\setup-bazel\0.8.5\dist\index.js:43:1)
      at setupBazel (C:\a\_actions\bazel-contrib\setup-bazel\0.8.5\dist\index.js:26:1)
      at run (C:\a\_actions\bazel-contrib\setup-bazel\0.8.5\dist\index.js:12:1)

I tried also with 1.20.0 instead of 1.x.
https://github.com/archonitelabs/radiant-cpp/actions/runs/9372780009/job/25805012776

  Error: Error: Unable to find Bazelisk version 1.20.0 for platform windows/arm64
      at downloadBazelisk (C:\a\_actions\bazel-contrib\setup-bazel\0.8.5\dist\index.js:90:1)
      at processTicksAndRejections (node:internal/process/task_queues:95:5)
      at setupBazelisk (C:\a\_actions\bazel-contrib\setup-bazel\0.8.5\dist\index.js:43:1)
      at setupBazel (C:\a\_actions\bazel-contrib\setup-bazel\0.8.5\dist\index.js:26:1)
      at run (C:\a\_actions\bazel-contrib\setup-bazel\0.8.5\dist\index.js:12:1)

@p0deje could this be related to?: bazelbuild/bazelisk#572

[p0deje]

Yes, it's related to this issue. If Bazelisk ships a native ARM64 binary, it would work out-of-the-box with setup-bazel.
We could work it around for now by forcing x64 architecture on Windows.

[jxy-s]

The drawback of forcing x64 bazelisk on Windows is that bazel won't use the right toolchain and will compile non-native x64 on ARM64, which is not desired. I think the best course of action is to resolve it in the bazelisk repo and produce an ARM64 binary for Windows.

[p0deje]

The drawback of forcing x64 bazelisk on Windows is that bazel won't use the right toolchain and will compile non-native x64 on ARM64, which is not desired. I think the best course of action is to resolve it in the bazelisk repo and produce an ARM64 binary for Windows.

Actually Bazel would still download proper toolchains and compilation should work correctly. I agree that it should be fixed upstream, but in the mean time - can you test p0deje/fix-windows-arm64 branch and let me know if it helps?

[jxy-s]

I am testing, but I'm confident that this is not going to do what is desired. Windows ARM64 using non-native bazel will build an x64 binary. In the case of a testing workflow that is very much undesired. You want to build and run tests on a native platform. I'll collect data and report back.

I suggest not merging #28 until we've come to a consensus here.

For reference: bazelbuild/bazel#22164 (comment) and bazelbuild/bazelisk#572

[jxy-s]

It seems like it's still broken for another reason? Looks like bazelisk-version: 1.x here is succeeding but the command can't locate the newly installed bazelisk? I tried specifying both bazel and bazelisk without luck.

https://github.com/archonitelabs/radiant-cpp/actions/runs/9451455948/job/26032443902#step:4:5

https://github.com/archonitelabs/radiant-cpp/actions/runs/9451414740/job/26032294653?pr=19#step:4:5

[p0deje]

Yes, I think there is an issue with how file is added to the tool cache, let me try to prepare another version.

[p0deje]

@jxy-s Can you please re-run with debug logs enabled and share here?

[jxy-s]

Here is a run with debug logs enabled: https://github.com/archonitelabs/radiant-cpp/actions/runs/9452727386/job/26037982135

[p0deje]

Maybe that's because .exe is missing from filename, can you try a latest commit?

[jxy-s]

https://github.com/archonitelabs/radiant-cpp/actions/runs/9454749231/job/26043007626?pr=21

Looks like that fixes the missing bazel (aka bazelisk) - however there is already a problem emerging with bazelisk 😄:

Run bazel test --test_output=all -s --toolchain_resolution_debug=.* //...
  bazel test --test_output=all -s --toolchain_resolution_debug=.* //...
  shell: C:\Windows\system3[2](https://github.com/archonitelabs/radiant-cpp/actions/runs/9454749231/job/26043007626?pr=21#step:4:2)\cmd.EXE /D /E:ON /V:OFF /S /C "CALL "{0}""
  
2024/06/10 20:01:20 Downloading https://releases.bazel.build/7.1.1/release/bazel-7.1.1-windows-x86_64.exe...
Error: Process completed with exit code -1073741515.

0xC0000135    STATUS_DLL_NOT_FOUND
{Unable To Locate Component} This application has failed to start because %hs was not found. Reinstalling the application might fix this problem.

Looking past that for a moment, bazelisk is not downloading the correct bazel. It is installing bazel-7.1.1-windows-x86_64.exe. That is going to do the wrong thing for an ARM64 Windows machine, explained here: bazelbuild/bazel#22164 (comment)

Anyway, all that aside. I think there are two things here.

1. bazelisk needs to provide a native ARM64 binary for Windows
2. setup-bazel might need to make a request to update tools for Windows ARM64 support: https://github.com/actions/runner-images

[p0deje]

1. Ok, agreed on that, let's not attempt to work it around in setup-bazel and fix it upstream in Bazelisk. It seems like a straightforward fix so hopefully the team can do it.
2. I actually don't follow how you used Windows ARM64 runner - I don't think GHA provides such a runner and you use something called archonite-windows-arm64 instead.

[jxy-s]

I actually don't follow how you used Windows ARM64 runner - I don't think GHA provides such a runner and you use something called archonite-windows-arm64 instead.

It is still a GitHub-hosted runner, it's just set up through the Archonite organization. The "Standard GitHub-hosted runners" does not yet include the ARM64 Windows or Linux machines in their set, but you can add them manually.

Link for information on this (and how to set them up) is in the original message in this thread: https://github.blog/2024-06-03-arm64-on-github-actions-powering-faster-more-efficient-build-systems/

[p0deje]

It is still a GitHub-hosted runner, it's just set up through the Archonite organization. The "Standard GitHub-hosted runners" does not yet include the ARM64 Windows or Linux machines in their set, but you can add them manually.

I see. This doesn't seem to be available to open-source projects yet, so you'd need to file the issue yourself.

[jxy-s]

You need to be looking at the organization level, I think you're looking at the project/repo level.

Configured here: https://github.com/bazel-contrib
Not here: https://github.com/bazel-contrib/setup-bazel

Anyway, it is available to open-source projects. Radiant is open source under the Archonite organization:
https://github.com/archonitelabs
https://github.com/archonitelabs/radiant-cpp

[p0deje]

The organization itself must be on Team / Enterprise plan, then it's available to its repositories:

[jxy-s]

Yeah, that tracks with the documentation. Soon enough they'll be available in the standard set for everyone.

Hopefully we can get a bit ahead of the curve here before more people run into issues.

[jxy-s]

@p0deje FYSA - Bazelisk now has a native arm64 binary for Windows available:
bazelbuild/bazel#22164
bazelbuild/bazelisk#601
https://github.com/bazelbuild/bazel/releases/tag/7.3.2

[p0deje]

@jxy-s Have you tried to use it on your ARM64 runners? No changes should be necessary on the action side to make it work.

[jxy-s]

I did and I got an error saying the bazel command was not found, despite specifying a bazelisk version:

  Windows-ARM64:
    runs-on: archonite-windows-arm64
    steps:
      - name: Checkout
        uses: actions/checkout@v4
      - name: Setup Bazel
        uses: bazel-contrib/[email protected]
        with:
          bazelisk-cache: true
          disk-cache: ${{ github.workflow }}
          repository-cache: true
          bazelisk-version: 1.x
      - name: Build and Run Unit Tests
        run: bazel test -s --toolchain_resolution_debug=.* //...
        shell: cmd

I got the same result specifying bazelisk instead of bazel as the command to run.

https://github.com/archonitelabs/radiant-cpp/actions/runs/11316873539/job/31469638751?pr=21

[p0deje]

@jxy-s Could you re-run with debug logs enabled and share them?

[jxy-s]

Here you go:
https://github.com/archonitelabs/radiant-cpp/actions/runs/11316873539/job/31470354919

   ##[debug]isExplicit: 
  ##[debug]explicit? false
  ##[debug]evaluating 0 versions
  ##[debug]match not found
  ##[debug]Attempting to download 1.x
  ##[debug]evaluating 30 versions
  ##[debug]matched: v1.22.0
  ##[debug]Downloading from https://github.com/bazelbuild/bazelisk/releases/download/v1.22.0/bazelisk-windows-arm64.exe
  ##[debug]Downloading https://github.com/bazelbuild/bazelisk/releases/download/v1.22.0/bazelisk-windows-arm64.exe
  ##[debug]Destination C:\a\_temp\31e08605-6d33-446f-a2aa-3d40e83f8281
  ##[debug]set auth
  ##[debug]download complete
  ##[debug]Adding to the cache...
  ##[debug]Caching tool bazelisk 1.x arm64
  ##[debug]source file: C:\a\_temp\31e08605-6d33-446f-a2aa-3d40e83f8281
  ##[debug]destination C:\a\_tool\bazelisk\1.x\arm64
  ##[debug]destination file C:\a\_tool\bazelisk\1.x\arm64\bazel
  ##[debug]finished caching tool
  ##[debug]Successfully cached bazelisk to C:\a\_tool\bazelisk\1.x\arm64

Looks like it's downloading it, but seems like maybe it's not being made available in the path? I could put in a hack to fix this by pointing at the location it downloads bazelisk. But maybe the setup should set up the path?

[p0deje]

Hmm, it should be as long as it's downloaded -

setup-bazel/index.js

Line 45 in 3736b3b

core.addPath(toolPath)

Can you see if bazelisk binary is available instead? Maybe additionally we need to expose bazel alias to bazelisk.

[jxy-s]

@p0deje sorry for the delay. I changed it from bazel to bazelisk and I'm getting the same result. Here are the logs with debugging enabled:

'bazelisk' is not recognized as an internal or external command,

https://github.com/archonitelabs/radiant-cpp/actions/runs/11533270059/job/32106080672

[p0deje]

@jxy-s Can you test if using dac93a5 fixes it for you? The downloaded binary should now be available as bazel or bazel.exe.

[jxy-s]

@p0deje unfortunately I get the same result. I tested both bazel and bazelisk:

https://github.com/archonitelabs/radiant-cpp/actions/runs/11605985370/job/32317389563

I'm starting to think this might be a bug in the way the workflow executes java on ARM64 on Windows. It might be another emulation oddity? If java is running x64 under emulation maybe the path isn't being updated/transferred to the child native ARM64 cmd.exe process?

[jxy-s]

Hm, actually - I'm not sure now... I specified the full path in this run and it still didn't find it:
https://github.com/archonitelabs/radiant-cpp/actions/runs/11606186526/job/32317843299

Setup Bazelisk
  ##[debug]isExplicit: 
  ##[debug]explicit? false
  ##[debug]evaluating 0 versions
  ##[debug]match not found
  ##[debug]Attempting to download 1.x
  ##[debug]evaluating 30 versions
  ##[debug]matched: v1.22.1
  ##[debug]Downloading from https://github.com/bazelbuild/bazelisk/releases/download/v1.22.1/bazelisk-windows-arm64.exe
  ##[debug]Downloading https://github.com/bazelbuild/bazelisk/releases/download/v1.22.1/bazelisk-windows-arm64.exe
  ##[debug]Destination C:\a\_temp\2369a7f8-e4cb-4abb-aea4-77852e2d9a6e
  ##[debug]set auth
  ##[debug]download complete
  ##[debug]Adding to the cache...
  ##[debug]Caching tool bazelisk 1.x arm64
  ##[debug]source file: C:\a\_temp\2369a7f8-e4cb-4abb-aea4-77852e2d9a6e
  ##[debug]destination C:\a\_tool\bazelisk\1.x\arm64
  ##[debug]destination file C:\a\_tool\bazelisk\1.x\arm64\bazel
  ##[debug]finished caching tool
  ##[debug]Successfully cached bazelisk to C:\a\_tool\bazelisk\1.x\arm64
  ::endgroup::
  
  
  ##[debug]Evaluating condition for step: 'Build and Run Unit Tests'
##[debug]Evaluating: success()
##[debug]Evaluating success:
##[debug]=> true
##[debug]Result: true
##[debug]Starting: Build and Run Unit Tests
##[debug]Loading inputs
##[debug]Loading env
Run C:\a\_tool\bazelisk\1.x\arm64\bazel.exe test -s --toolchain_resolution_debug=.* //...
  
##[debug]C:\Windows\system32\cmd.EXE /D /E:ON /V:OFF /S /C "CALL "C:\a\_temp\25222223-55dc-4f73-9600-788e44e92545.cmd""
'C:\a\_tool\bazelisk\1.x\arm64\bazel.exe' is not recognized as an internal or external command,
operable program or batch file.
Error: Process completed with exit code 1.
##[debug]Finishing: Build and Run Unit Tests

[p0deje]

Hmm, can you list the directory and see files it has?

[jxy-s]

Looks like it's still writing it without the .exe extension?:
https://github.com/archonitelabs/radiant-cpp/actions/runs/11639690161/job/32416168573?pr=21

Run bazel-contrib/setup-bazel@dac93a5ea3e0f98fc0d5b3a4f7700f579b96612d
...
Setup Bazelisk
  ##[debug]isExplicit: 
  ##[debug]explicit? false
  ##[debug]evaluating 0 versions
  ##[debug]match not found
  ##[debug]Attempting to download 1.x
  ##[debug]evaluating 30 versions
  ##[debug]matched: v1.22.1
  ##[debug]Downloading from https://github.com/bazelbuild/bazelisk/releases/download/v1.22.1/bazelisk-windows-arm64.exe
  ##[debug]Downloading https://github.com/bazelbuild/bazelisk/releases/download/v1.22.1/bazelisk-windows-arm64.exe
  ##[debug]Destination C:\a\_temp\0f418475-e061-40ba-8655-854e11f5abd4
  ##[debug]set auth
  ##[debug]download complete
  ##[debug]Adding to the cache...
  ##[debug]Caching tool bazelisk 1.x arm64
  ##[debug]source file: C:\a\_temp\0f418475-e061-40ba-8655-854e11f5abd4
  ##[debug]destination C:\a\_tool\bazelisk\1.x\arm64
  ##[debug]destination file C:\a\_tool\bazelisk\1.x\arm64\bazel
  ##[debug]finished caching tool
  ##[debug]Successfully cached bazelisk to C:\a\_tool\bazelisk\1.x\arm64
...
Run dir C:\a\_tool\bazelisk\1.x\arm64 && C:\a\_tool\bazelisk\1.x\arm64\bazel.exe test -s --toolchain_resolution_debug=.* //...
##[debug]C:\Windows\system32\cmd.EXE /D /E:ON /V:OFF /S /C "CALL "C:\a\_temp\0d29e1a1-9f20-40bd-9249-86234d1f4543.cmd""
 Volume in drive C is Windows
 Volume Serial Number is 10AA-CE08
 Directory of C:\a\_tool\bazelisk\1.x\arm64
11/02/2024  04:05 AM    <DIR>          .
11/02/2024  04:05 AM    <DIR>          ..
11/02/2024  04:05 AM         8,598,016 bazel
               1 File(s)      8,598,016 bytes
               2 Dir(s)  108,558,503,936 bytes free
'C:\a\_tool\bazelisk\1.x\arm64\bazel.exe' is not recognized as an internal or external command,
operable program or batch file.
Error: Process completed with exit code 1.
##[debug]Finishing: Build and Run Unit Tests

[p0deje]

@jxy-s Hmm, can you try using an exact version of Bazelisk in bazelisk-version? I wonder if the tool-cache somehow loads the previous downloads before I added .exe extension.