Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Env Detection #507

Merged
merged 25 commits into from
Oct 30, 2023
Merged

Env Detection #507

merged 25 commits into from
Oct 30, 2023

Conversation

JonathanHenson
Copy link
Contributor

@JonathanHenson JonathanHenson commented Oct 7, 2023

For p4d.24xlarge:

Is on Ec2 Instance nitro ? True
Instance Type = p4d.24xlarge
Is Optimized for System = True

For trn1n.32xlarge:

Is on Ec2 Instance nitro ? True
Instance Type = trn1n.32xlarge
Is Optimized for System =  True

For p5.48xlarge:

Is on Ec2 Instance nitro ? True
Instance Type = p5.48xlarge
Is Optimized for System = True

For c5n.18xlarge:

Is on Ec2 Instance nitro ? True
Instance Type = c5n.18xlarge
Is Optimized for System = False

For t2.xlarge

Is on Ec2 Instance nitro ? {} False
Instance Type = {} None
Is Optimized for System = {} False

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

awscrt/common.py Outdated
class SystemEnvironment:

def __init__(self):
self._env = _awscrt.load_system_environment()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

trivial/style: we always call this variable _binding

Suggested change
self._env = _awscrt.load_system_environment()
self._binding = _awscrt.load_system_environment()

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just going to pull it and cache internally

source/common.c Outdated
Comment on lines 33 to 36
assert(PyCapsule_CheckExact(sys_env_capsule));

struct aws_system_environment *env = PyCapsule_GetPointer(sys_env_capsule, s_capsule_name_sys_env);
assert(env);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

trivial: I've never figured out how to make python compile its C-extensions in debug mode, so these asserts are pointless

remove them, or use AWS_FATAL_ASSERT

source/common.c Outdated Show resolved Hide resolved
source/common.c Outdated Show resolved Hide resolved
source/common.c Outdated Show resolved Hide resolved
source/common.c Outdated Show resolved Hide resolved
source/module.c Outdated
Comment on lines 673 to 674
AWS_PY_METHOD_DEF(load_system_environment, METH_NOARGS),
AWS_PY_METHOD_DEF(is_env_ec2, METH_VARARGS),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I got confused when the function names changed at the different layers. We usually have them match the underlying C function

Suggested change
AWS_PY_METHOD_DEF(load_system_environment, METH_NOARGS),
AWS_PY_METHOD_DEF(is_env_ec2, METH_VARARGS),
AWS_PY_METHOD_DEF(system_environment_load, METH_NOARGS),
AWS_PY_METHOD_DEF(system_environment_is_running_on_ec2, METH_VARARGS),

awscrt/common.py Outdated
Comment on lines 32 to 33
def is_ec2_nitro_instance(self) -> bool:
return _awscrt.is_env_ec2(self._env)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why mention "nitro" here, but not in the underlying C function?

I liked the is_running_on_ec2 name better, FWIW

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's really a nitro check, and ec2 is implied as a result.

I'll make sure nitro is put in there and fuse it with the suggestion you made

awscrt/common.py Outdated
def get_ec2_instance_type(self) -> str:
return self._detected_instance_type

def is_crt_s3_optimized_for_system_env(self) -> bool:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is awkward:
system_env.is_crt_s3_optimized_for_system_env()

flip the words around to sound more natural:
system_env.is_optimized_for_crt_s3()

or it might feel more natural as a separate function in s3.py:
awscrt.s3.is_optimized_for_system(system_env)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or just have 1 public function: awscrt.s3.is_optimized_for_system()

and don't bother binding the aws_system_environment at all, just keep it as an implementation detail

we can have python cache the answer, so it only does work the first time it's called

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's fine. I added verbosity here largely for aid in debugging when something goes wrong, we can at least ask the customer to add a python function to their script and it's much simpler to track down, than having to peel through logs or c source.

awscrt/common.py Outdated
def is_ec2_nitro_instance(self) -> bool:
return _awscrt.is_env_ec2(self._env)

def get_ec2_instance_type(self) -> str:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the underylying C function is get_virtualization_product_name()

let's consistently call it "ec2" or "virtualization_product_name" but it's annoying to change terminology between the layers

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ec2 and vritualization product name are not the same thing. ec2 is A vendor, and aws-c-common is mostly generic, I did move most of this to aws-c-s3 however and it makes more sense there. I can clean it up here.

awscrt/common.py Outdated
Comment on lines 22 to 24
class SystemEnvironment:

def __init__(self):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Inherit from NativeResource. we just do this to help tests track leaks

Suggested change
class SystemEnvironment:
def __init__(self):
class SystemEnvironment(NativeResource):
def __init__(self):
super().__init__()

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm just gonna pull out systemenvironment entirely and cache it internally as you suggested above.

JonathanHenson and others added 10 commits October 10, 2023 10:42
PR trivial merge

Co-authored-by: Michael Graeb <[email protected]>
PR trivial merge

Co-authored-by: Michael Graeb <[email protected]>
PR trivial merge

Co-authored-by: Michael Graeb <[email protected]>
PR trivial merge

Co-authored-by: Michael Graeb <[email protected]>
PR trivial merge

Co-authored-by: Michael Graeb <[email protected]>
PR trivial merge

Co-authored-by: Michael Graeb <[email protected]>
PR trivial merge

Co-authored-by: Michael Graeb <[email protected]>
source/common.c Outdated Show resolved Hide resolved
@waahm7 waahm7 changed the title Env detection, from python crt. Do not merge yet. Env Detection Oct 27, 2023
awscrt/s3.py Outdated Show resolved Hide resolved
continuous-delivery/test-pip-install.py Outdated Show resolved Hide resolved
@waahm7 waahm7 enabled auto-merge (squash) October 30, 2023 15:35
@waahm7 waahm7 merged commit d62491f into main Oct 30, 2023
@waahm7 waahm7 deleted the env_detection branch October 30, 2023 15:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants