Using ENTRYPOINT in custom Docker image overrules the "container start command" in the template of a deployed endpoint

[TimPietrusky]

In order to develop a custom worker using runpodctl project dev, I needed to overwrite Dockerfile using runpod/base:0.6.1-cuda12.2.0 as the base. When using CMD /start.sh in my custom Dockerfile, everything works as expected. When using ENTRYPOINT ["/start.sh"] the worker also works during development (e.g. runpodctl project dev), but fails during deployment as a serverless endpoint (e.g. runpodctl project deploy). It seems that the ENTRYPOINT overwrites the container start command in the template that is generated after running runpodctl project deploy.

Steps to reproduce

1. Clone this repository (which is just using the "Hello World" example from runpodctl project create + runpodctl project build and replacing CMD python handler.py in the generated Dockerfile with ENTRYPOINT /setup.sh and using the resulting image as the base image in runpod.toml)
2. Run runpodctl project deploy
3. Create a request to the deployed endpoint:

curl -X POST https://api.runpod.ai/v2/ENDPOINT_ID/runsync \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer RUNPOD_API_KEY" \
  -d '{"input": {"name": "world"}}'

4. See that the request keeps for every in state IN_QUEUE
5. See in the container logs, that the worker started nginx and fails to start ssh. Both of which should not be started, as the container start command in the template should overwrite this.

2024-10-10T14:49:53.404735708Z Starting Nginx service...
2024-10-10T14:49:53.419883638Z  * Starting nginx nginx
2024-10-10T14:49:53.429388613Z    ...done.
2024-10-10T14:49:53.429921329Z Pod Started
2024-10-10T14:49:53.429939970Z Exporting environment variables...
2024-10-10T14:49:53.434187082Z Running post-start script...
2024-10-10T14:49:53.436783092Z Monitoring SSH connections every 60 seconds, with a countdown of 120 seconds.
2024-10-10T14:49:53.437149813Z Start script(s) finished, pod is ready to use.
2024-10-10T14:49:53.444669267Z 2024/10/10 14:49:53 No config file used
2024-10-10T14:49:53.510740677Z 2024/10/10 14:49:53 Listening on [::]:4040
2024-10-10T14:50:53.489170777Z No SSH connections found. Countdown: 60 seconds remaining.
2024-10-10T14:51:53.518503470Z No SSH connections found. Countdown: 0 seconds remaining.
2024-10-10T14:51:53.518539851Z Countdown reached zero. Removing pod: 7t8qem4e23vnvn
2024-10-10T14:51:53.657219123Z Runpod config file not found, please run `runpodctl config` to create it
2024-10-10T14:51:53.658047279Z API key not found
2024-10-10T14:51:53.658095471Z Error: API key not found

Expected

• It either works out of the box OR is at least documented to not be working when using ENTRYPOINT
• runpodctl project deploy could also check the Dockerfile and provide an info to the user, that "deploy" is not working when an ENTRYPOINT is used instead of CMD (if this problem can't be fixed)

Workaround

Don't use ENTRYPOINT, but CMD in your custom Dockerfile.