[FR]: Enable overwriting parameters in ads opctl run for Data Science Jobs, Pipelines and Data Flow Applications.

[mrDzurb]

Willingness to contribute

Yes. I can contribute this feature independently.

Proposal Summary

Currently, the "ads opctl" command-line tool allows users to run existing Data Science jobs, pipelines and Data Flow applications by providing their OCIDs. However the tool lacks the capability to overwrite certain parameters associated with infrastructure or runtime settings.

This feature request aims to enhance the "ads opctl run" functionality by introducing the ability to overwrite parameters when running these jobs, applications or pipelines. By allowing users to modify infrastructure and runtime-related parameters, such as resource allocation, compute configurations, or environment variables, it would provide greater flexibility and control over the execution process.

Motivation

With the proposed changes, users will be able to specify custom parameter values the supersede the default or pre-configured settings during runtime. This will enable them to tailor the infrastructure and runtime aspects according to their specific requirements, ensuring optimal performance and resource utilization.

The ability to overwrite parameters will empower data scientists, data engineers, and other users of the ads tool to experiment, fine-tune, and optimize their workloads without having to create new job instances or modify existing configurations. This enhanced flexibility will significantly improve the productivity and efficiency of data-centric operations.

By implementing this feature, the "ads opctl run" command will become more versatile and adaptable to various scenarios, enabling seamless integration into different data processing pipelines and workflows.

Details

The enhancement could enable users to define an configuration YAML file that contains parameter values to the overridden during runtime. This approach simplifies the command execution process and allows for easier management and sharing of parameter configurations.

Example:

ads opctl run --ocid ocid1.datasciencejob.oc1... --file path/to/the/YAML