Kfp workflow for html2parquet #694
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
revit13
force-pushed
the
kfp-html2parquet
branch
4 times, most recently
from
b2d9ae9 to
d77c2ba
Compare
roytman
reviewed
Oct 13, 2024
revit13
force-pushed
the
kfp-html2parquet
branch
4 times, most recently
from
14d29fd to
a4a0778
Compare
Signed-off-by: Revital Sur <[email protected]>
Signed-off-by: Revital Sur <[email protected]>
Signed-off-by: Revital Sur <[email protected]> Co-authored-by: Alexey Roytman <[email protected]>
Signed-off-by: Revital Sur <[email protected]>
Signed-off-by: Revital Sur <[email protected]>
Signed-off-by: Revital Sur <[email protected]>
Signed-off-by: Revital Sur <[email protected]>
Signed-off-by: Revital Sur <[email protected]>
Signed-off-by: Revital Sur <[email protected]>
Signed-off-by: Revital Sur <[email protected]>
Signed-off-by: Revital Sur <[email protected]>
Signed-off-by: Revital Sur <[email protected]>
revit13
force-pushed
the
kfp-html2parquet
branch
from
a4a0778 to
ba3c1c2
Compare
Done. Thanks |
revit13
force-pushed
the
kfp-html2parquet
branch
from
ba3c1c2 to
a71c670
Compare
Signed-off-by: Revital Sur <[email protected]>
roytman
reviewed
Oct 15, 2024
| @@ -103,8 +105,13 @@ def {{ pipeline_name }}( | |||
| # Ray cluster | |||
| ray_name: str = "{{ pipeline_name }}-kfp-ray", # name of Ray cluster | |||
| # Add image_pull_secret and image_pull_policy to ray workers if needed | |||
| {%- if image_pull_secret != "" %} | |||
There was a problem hiding this comment.
can we reuse the same dictionary and just add image_pull_secret when it is defined?
transforms/add_new_kfp_workflow.md
Outdated
|
|
||
| 1) Create the workflow. [pipeline generator](../kfp/pipeline_generator/single-pipeline/) can be used for that. | ||
|
|
||
| 2) Create a new `kfp_ray` directory in the transform directory, similar to [this directory](universal/noop/kfp_ray/). Inside this directory, include the following: |
There was a problem hiding this comment.
should we replace the steps: create a directory, and after that add a template and generate the pipeline?
| pipeline_arguments: | ||
| - name: data_files_to_use | ||
| type: "str" | ||
| value: "['.html', '.zip']" |
There was a problem hiding this comment.
does the transformer support zip files?
There was a problem hiding this comment.
Signed-off-by: Revital Sur <[email protected]>
revit13
force-pushed
the
kfp-html2parquet
branch
from
f6eee4c to
5419cd1
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why are these changes needed?
Add kfp workflow to html2parquet
Related issue number (if any).
issue #690