outputFiles drops folder structure

[guslipkin]

Describe the issue

If I have a task that creates a file with a specific folder structure, such as an Arrow dataset, then the intermediate folder structure is dropped and only the file itself is made available in later tasks. Sometimes that folder structure is very important and it should be kept.

A discussion about this in the Kestra Slack can be found here: https://kestra-io.slack.com/archives/C03FQKXRK3K/p1736048596068889

For example, here we save a file to tmp/cyl={{taskrun.value}}/part-0.parquet. However, in Kestra, that path becomes kestra:///path/random_folder_names/random-prefix-part0.parquet.

id: myflow-copy
namespace: company.team

tasks:
  - id: query_parallel
    type: io.kestra.plugin.core.flow.ForEach
    values: "[4,6,8]"
    concurrencyLimit: 0
    tasks:
      - id: query
        type: io.kestra.plugin.scripts.shell.Commands
        taskRunner:
          type: io.kestra.plugin.scripts.runner.docker.Docker
        namespaceFiles:
          enabled: true
        commands:
          - mkdir -p tmp/cyl={{taskrun.value}}
          - touch tmp/cyl={{taskrun.value}}/part-0.parquet
        outputFiles:
          - "tmp/**/*.parquet"

  - id: log_combine_results
    type: io.kestra.plugin.core.log.Log
    message: "{{ outputs.query }}"

Environment

• Kestra Version: open source