Add Support for Google Cloud Run Job Executor

[brofori]

What's the use case?

Integrating Cloud Run as a job executor in Dagster would enable users to leverage these benefits for their data pipelines. Especially for compute intensive long running jobs.

Ideas of implementation

• Develop a new executor within Dagster that interfaces with the Google Cloud Run SDK to run cloud run jobs. (Similar to ecs_executor)
• Provide configuration options for users to specify Cloud Run settings, such as job names, regions, and resource allocations.

Additional information

I believe that adding support for Google Cloud Run as a job executor would significantly enhance Dagster’s capabilities by providing a serverless execution option. I am willing to contribute to the development of this feature and can assist with implementation and testing.

Message from the maintainers

Impacted by this issue? Give it a 👍! We factor engagement into prioritization.

[garethbrickman]

May be worth linking this in this Discussion where there are other interested users: #18958

Additionally, there is this community integration for a Google Cloud Run run_launcher: https://github.com/dagster-io/community-integrations/tree/main/libraries/dagster-contrib-gcp

[mkleinbort-wl]

THIS FEATURE! YES, PLEASE. (The all caps are intentional).

Azure Container Apps executor would be amazing as well.

As I have mentioned across Slack and other issues - scaling Dagster horizontally is extremely cumbersome, with only the Kubernetes deployment or the Ray executor as viable alternatives at present.