Assume AWS role via EC2 instance identity / IMDSv2 made difficult

[roehrijn]

The AWS provider allows to make use of EC2 instance identity through the AWS_INSTANCE_PROFILE_ARN configuration option. However, in

devpod-provider-aws/pkg/aws/aws.go

Line 775 in 566b812

HttpPutResponseHopLimit: aws.Int32(1),

the hop limit is set to 1 instead of keeping the default. This prevents AWS workloads running inside the devcontainer setup from fetching a machine identity based token via IMDSv2. Thus, it's for example not possible to use the AWS Cli from within the container. One can configure the required roles/profile, but afaik it is not possible to use it.
That's why I want to bring up the question: Is this implemented like that by purpose or maybe just a mistake? If it is the latter, I'm eager to contribute a change but the question would be, should we omit this line completely and stick to defaults, which can be preconfigured on a per-AWS-account level. Or should we add a config option to override that or set it to 2 as long as AWS_INSTANCE_PROFILE_ARN is set? Or both / all of them?

WDYT?

_{Jan Roehrich [email protected], Mercedes-Benz Tech Innovation GmbH, legal info/Impressum}

[shanman190]

Hi @roehrijn, I'll chime in a little bit with some of the background context while we wait for a core contributor to provide some feedback as well.

So the providers are used in two possible ways:

• Directly from the host machine of the user (OSS)
• Via the Pro products (PAID)

When discussing this with @89luca89 and @pascalbreuninger, the identity used should really be of the end user rather than a system identity (as it hides the user who performed the actions). So it felt more secure to only allow the DevPod agent -- which runs inside the AWS instance -- to use the instance profile, then to expect the user to provide credentials for their user identity when working inside of the container workspace. The DevPod agent itself only needs to have the ability to stop the EC2 instance that it is running within which is also why the permissions were reigned in with the commits in which this change was introduced.

[roehrijn]

Hi @shanman190, thanks for the insights. I totally agree about the character of the machine identity. However, there is currently a lack of support for using OIDC handshakes with CLI tools within the devcontainer. That for example makes use of Hashicorp Vault CLI in the container not so nice. That's why we have machine identity in mind.
However, the usecase we currently think of, is for the time being just the lookup of a certain S3 bucket which we need as a build cache. This can also be solved by implementing a lookup via DNS SRV record.

But what would you folks think about implementing a config option for the hop limit with a default of 1? I'm eager to add that to the provider. Maybe together with a small disclaimer about security in the config option description?

[shanman190]

@roehrijn, I'll double check this, but I'm pretty sure that the OIDC stuff should work just fine. It all just depends on the CLI's ephemeral port being forwarded from the devcontainer to the host machine, then the interaction occurs from the host machine. The port forward itself may not be automatic, so there may be a user click necessary to connect the port, but I'll check this out and post back here with what I find.

As for the hop limit question, I personally could go either way. Some users may not care about using the machine identity while others do. I do still feel that the user identity is better than the machine identity when I look at it solely from an AWS standpoint. We'd have to see what the maintainers think (I'm just an avid contributor as well). Like I mentioned earlier though, I'm pretty sure that I've used OIDC successfully even in this context already for the CLI tools that I interact with, so I'll drop an update here once I've had a chance to double check that.

[pascalbreuninger]

@roehrijn @shanman190 AWS is always a bit foreign to me, thanks for stepping in and helping out.
I can see valid points for both ways. To me it comes down to the question if the people using the workspace would've had permissions to manage the EC2 instance anyway.
If that's the case then using the instance identity is straightforward and should be allowed.
If not, having users sign in with their actual identity is the way to go.

For other providers we've chosen the route of "open by default" and added options to tighten security through options. This ensures most people using DevPod OSS will have a working experience out of the box and enterprises, usually more on DevPod Pro, can still get the restrictions in place they need to run workspaces securely.
What do you guys think?

Aside: The whole OIDC flow should work out of the box without people having to manually forward the port from their IDE just to authenticate. I'll look into ways of making that possible as pretty much every CLI would benefit from it

[shanman190]

Chatted with @pascalbreuninger a little bit about this.

Let's add a parameter to allow adjusting the hop limit on the created EC2 instance.

The idea being that we should carry a secure by default (hop limit=1; just enough AWS permissions for the DevPod agent to function), then the user should be able to allow the instance profile to be used inside of the workspaces if they so choose. This follows a similar pattern as other providers.

Additionally, we both agreed that OIDC workflows should probably be better overall for all providers, so he was going to look into that a little bit more.

So I think the todo for this issue is to add the hop limit option so that the end user can configure it to be higher than the default of 1.