Not authenticated to use blobs outside of Azure blob container working directory when using Azure Entra

[adamrtalbot]

Related to #5448 and #5444 but both issues refer to using Fusion, this one refers to using azcopy.

They are likely to be solved by the same method, since they have the same underlying challenge: how to pass authentication to the worker node (Batch) from Nextflow.

I seem to be able to recreate the issue without Fusion.

> nextflow run seqeralabs/nf-canary -r main --remoteFile az://igenomes/atacseq_samplesheet_custom.csv --run TEST_STAGE_REMOTE -w az://scidev-useast -c azure.config
N E X T F L O W  ~  version 24.10.3
NOTE: Your local project version looks outdated - a different revision is available in the remote repository [c818260035]
Launching `https://github.com/seqeralabs/nf-canary` [magical_noyce] DSL2 - revision: 2ad4214f51 [main]
Uploading local `bin` scripts folder to az://scidev-useast/tmp/cf/bcc6a54f6a9dd33780a5251d956439/bin
[69/6f65a5] Submitted process > NF_CANARY:TEST_STAGE_REMOTE (1)
ERROR ~ Error executing process > 'NF_CANARY:TEST_STAGE_REMOTE (1)'

Caused by:
  Process `NF_CANARY:TEST_STAGE_REMOTE (1)` terminated with an error exit status (1)


Command executed:

  cat atacseq_samplesheet_custom.csv

Command exit status:
  1

Command output:
  (empty)

Work dir:
  az://scidev-useast/69/6f65a5549f7a3b2357312b12a28996

Container:
  docker.io/library/ubuntu:23.10

Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named `.command.sh`

 -- Check '.nextflow.log' file for details
Execution cancelled -- Finishing pending tasks before exit

azure.config:

process.executor = 'azurebatch'

fusion {
    enabled = false
}

azure {

    storage {
        accountName = 'seqeralabs'
    }

    batch {
        location = 'eastus'
        accountName = 'seqeralabs'
        copyToolInstallMode = 'node'
        autoPoolMode = true
        allowPoolCreation = true
        deletePoolsOnCompletion = false
    }

    activeDirectory {
        servicePrincipalId = 'redacted'
        servicePrincipalSecret = 'redacted'
        tenantId = 'redacted'
    }
}

And with an access key:

To reiterate what's been said above, the error appears to stem from generateContainerSasWithActiveDirectory, which is only generating a relevant key for the working container and nothing else. Generating an account level SAS seems tricky (according to @alberto-miranda).

Originally posted by @adamrtalbot in #5444 (comment)

[adamrtalbot]

@alberto-miranda here is a method we could tell nextflow to pass the details to the worker task, this could help with #5444 and #5448.

It's pretty crude right now.