Skip to content

Commit

Permalink
Merge branch 'dev' into missing-ray-notebook
Browse files Browse the repository at this point in the history
  • Loading branch information
touma-I committed Jan 14, 2025
2 parents 2ff7ec7 + 2b19b7f commit 557e58a
Show file tree
Hide file tree
Showing 70 changed files with 854 additions and 8,948 deletions.
1 change: 1 addition & 0 deletions .make.defaults
Original file line number Diff line number Diff line change
Expand Up @@ -226,6 +226,7 @@ __check_defined = \
--platform $(DOCKER_PLATFORM) \
--build-arg EXTRA_INDEX_URL=$(EXTRA_INDEX_URL) \
--build-arg BASE_IMAGE=$(BASE_IMAGE) \
--build-arg TRANSFORM_NAME=$(TRANSFORM_NAME) \
--build-arg DPK_WHEEL_FILE_NAME=$(DPK_WHEEL_FILE_NAME) \
--build-arg BUILD_DATE=$(shell date -u +'%Y-%m-%dT%H:%M:%SZ') \
--build-arg GIT_COMMIT=$(shell git log -1 --format=%h) .
Expand Down
2 changes: 1 addition & 1 deletion examples/kfp-pipelines/superworkflows/ray/kfp_v2/Makefile
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
REPOROOT=${CURDIR}/../../../..
REPOROOT=${CURDIR}/../../../../..
WORKFLOW_VENV_ACTIVATE=${REPOROOT}/transforms/venv/bin/activate
include $(REPOROOT)/transforms/.make.workflows

Expand Down
6 changes: 3 additions & 3 deletions examples/kfp-pipelines/superworkflows/ray/kfp_v2/README.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
# Chaining transforms using KFP V2

As in [super pipelines of KFP v1](../../../doc/multi_transform_pipeline.md), we want to offer an option of running a series of transforms one after the other on the data. But, in KFP v2 we can make it easier to chain transforms using the [nested pipelines](https://www.kubeflow.org/docs/components/pipelines/user-guides/components/compose-components-into-pipelines/#pipelines-as-components) that KFP v2 offers.
As in [super pipelines of KFP v1](../../../../../kfp/doc/multi_transform_pipeline.md), we want to offer an option of running a series of transforms one after the other on the data. But, in KFP v2 we can make it easier to chain transforms using the [nested pipelines](https://www.kubeflow.org/docs/components/pipelines/user-guides/components/compose-components-into-pipelines/#pipelines-as-components) that KFP v2 offers.

One example of chaining `noop` and `document id` transforms can be found [here](superpipeline_noop_docId_v2.py). When running this pipeline it appears as hierarchical graph with two nested pipelines, one for each transform as shown in the following screenshots.
One example of chaining `noop` and `document id` transforms can be found [here](superpipeline_noop_docId_v2_wf.py). When running this pipeline it appears as hierarchical graph with two nested pipelines, one for each transform as shown in the following screenshots.

`root` Layer
![nested_pipeline](nested_pipeline.png)
Expand All @@ -27,6 +27,6 @@ Another useful feature of the KFP v2 is the `Json` editor for the `dict` type in
cd examples/kfp/superworkflows/ray/kfp_v2/
make clean
export KFPv2=1
export PYTHONPATH=../../../../transforms
export PYTHONPATH=../../../../../transforms
make workflow-build
```
4 changes: 0 additions & 4 deletions examples/notebooks/archive/README.md

This file was deleted.

43 changes: 0 additions & 43 deletions examples/notebooks/archive/code/README.md

This file was deleted.

2,345 changes: 0 additions & 2,345 deletions examples/notebooks/archive/code/demo_with_apis.ipynb

This file was deleted.

Loading

0 comments on commit 557e58a

Please sign in to comment.