Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tasks aren't able to save files to local storage on develop #6712

Open
wrussell1999 opened this issue Jan 10, 2025 · 0 comments
Open

Tasks aren't able to save files to local storage on develop #6712

wrussell1999 opened this issue Jan 10, 2025 · 0 comments
Labels
area/backend Needs backend code changes area/frontend Needs frontend code changes bug Something isn't working

Comments

@wrussell1999
Copy link
Member

wrussell1999 commented Jan 10, 2025

Describe the issue

Students from the DTC DE Zoomcamp course are reporting issues where their execution fails because expected files were not generated as output files.

Error from student (because file didn't exist):
image

Screenshot 2025-01-10 at 16 23 26

No Output Files are generated:

Screenshot 2025-01-10 at 16 33 35

Specifically from the extract task of type io.kestra.plugin.scripts.shell.Commands

id: postgres_taxi
namespace: zoomcamp
description: |
  The CSV Data used in the course: https://github.com/DataTalksClub/nyc-tlc-data/releases

inputs:
  - id: taxi
    type: SELECT
    displayName: Select taxi type
    values: [yellow, green]
    defaults: yellow
  - id: year
    type: SELECT
    displayName: Select year
    values: ["2019", "2020"]
    defaults: "2019"
  - id: month
    type: SELECT
    displayName: Select month
    values: ["01", "02", "03", "04", "05", "06", "07", "08", "09", "10", "11", "12"]
    defaults: "01"

variables:
  file: "{{inputs.taxi}}_tripdata_{{inputs.year}}-{{inputs.month}}.csv"
  staging_table: "public.{{inputs.taxi}}_tripdata_staging"
  table: "public.{{inputs.taxi}}_tripdata"
  # data: green_tripdata_2019-01.csv
  data: "{{outputs.extract.outputFiles[inputs.taxi ~ '_tripdata_' ~ inputs.year ~ '-' ~ inputs.month ~ '.csv']}}"

tasks:
  - id: set_label
    type: io.kestra.plugin.core.execution.Labels
    labels:
      file: "{{render(vars.file)}}"
      taxi: "{{inputs.taxi}}"
  - id: extract
    warningOnStdErr: false
    type: io.kestra.plugin.scripts.shell.Commands
    outputFiles:
      - "*.csv"
    taskRunner:
      type: io.kestra.plugin.core.runner.Process
    commands:
      - wget -qO- https://github.com/DataTalksClub/nyc-tlc-data/releases/download/{{inputs.taxi}}/{{render(vars.file)}}.gz | gunzip > {{render(vars.file)}}
  - id: log
    type: io.kestra.plugin.core.log.Log
    message: "{{ render(vars.data) }}"  

Environment

  • Kestra Version: develop
@wrussell1999 wrussell1999 added bug Something isn't working area/backend Needs backend code changes area/frontend Needs frontend code changes labels Jan 10, 2025
@github-project-automation github-project-automation bot moved this to Backlog in Issues Jan 10, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/backend Needs backend code changes area/frontend Needs frontend code changes bug Something isn't working
Projects
Status: Backlog
Development

No branches or pull requests

1 participant