From fd55b832765c14f9ebe3a7df6bc6baed3e7baece Mon Sep 17 00:00:00 2001 From: Stephanie Brink Date: Thu, 21 Mar 2024 11:30:59 -0700 Subject: [PATCH 01/13] initial commit on caliper modifier --- docs/3-opt-edit-experiment.rst | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/docs/3-opt-edit-experiment.rst b/docs/3-opt-edit-experiment.rst index e35ca1758..732ccf299 100644 --- a/docs/3-opt-edit-experiment.rst +++ b/docs/3-opt-edit-experiment.rst @@ -54,3 +54,11 @@ If you would like to modify a specification of your benchmark, you can do so by upstreaming changes to Spack and/or Ramble, or working on your benchmark specification in ``benchpark/repo/${BENCHMARK}`` (see :doc:`add-a-benchmark` for details). + +Caliper modifier +---------------- + +caliper docs +ramble modifier docs + +directions to turn on caliper modifier From b330c93de7c7a2ab95a4b8a2fc7cea042ef80e42 Mon Sep 17 00:00:00 2001 From: Stephanie Brink Date: Fri, 22 Mar 2024 10:02:35 -0700 Subject: [PATCH 02/13] add links --- docs/3-opt-edit-experiment.rst | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/docs/3-opt-edit-experiment.rst b/docs/3-opt-edit-experiment.rst index 732ccf299..8752ee1b0 100644 --- a/docs/3-opt-edit-experiment.rst +++ b/docs/3-opt-edit-experiment.rst @@ -58,7 +58,10 @@ or working on your benchmark specification in ``benchpark/repo/${BENCHMARK}`` Caliper modifier ---------------- -caliper docs -ramble modifier docs +Documentation on `Caliper `_ +Documentation on `Ramble Modifier `_ -directions to turn on caliper modifier +To turn on the caliper modifier, add ``--modifier=caliper`` to the Benchpark +setup step:: + + ./benchpark setup benchmark/programmingmodel system --modifier=caliper From aca0048d41a00248f320f25719d5e629abc1e1ac Mon Sep 17 00:00:00 2001 From: Stephanie Brink Date: Tue, 26 Mar 2024 14:40:44 -0700 Subject: [PATCH 03/13] updates --- docs/3-opt-edit-experiment.rst | 27 +++++++++++++++++++++------ 1 file changed, 21 insertions(+), 6 deletions(-) diff --git a/docs/3-opt-edit-experiment.rst b/docs/3-opt-edit-experiment.rst index 8752ee1b0..155bfd6a8 100644 --- a/docs/3-opt-edit-experiment.rst +++ b/docs/3-opt-edit-experiment.rst @@ -55,13 +55,28 @@ you can do so by upstreaming changes to Spack and/or Ramble, or working on your benchmark specification in ``benchpark/repo/${BENCHMARK}`` (see :doc:`add-a-benchmark` for details). -Caliper modifier ----------------- +Applying the Caliper modifier +----------------------------- +In Benchpark, a modifier follows `Ramble Modifier +`_ +and is an abstract object that can be applied to a large set of reproducible +specifications, such as injecting performance analysis. In Benchpark, we have +implemented a Caliper modifier to build the specifications with the +instrumentation and profiling library, Caliper, in a platform-independent +manner. With the modifier enabled, Caliper is configured to use the SPOT +built-in configuration, which records a time profile. More documentation +on Caliper can be found `here `_. -Documentation on `Caliper `_ -Documentation on `Ramble Modifier `_ - -To turn on the caliper modifier, add ``--modifier=caliper`` to the Benchpark +To turn on the Caliper modifier, add ``--modifier=caliper`` to the Benchpark setup step:: ./benchpark setup benchmark/programmingmodel system --modifier=caliper + + +After the experiments in the workspace have completed running, a ``.cali`` file +is created which contains the time metrics: + +* Min time/rank: Minimum time (in seconds) across all ranks +* Max time/rank: Maximum time (in seconds) across all ranks +* Avg time/rank: Average time (in seconds) across all ranks +* Total time: Aggregated time (in seconds) over all ranks From b41c5394eace58050245cf91e0fdab013c9a0cc9 Mon Sep 17 00:00:00 2001 From: Stephanie Brink Date: Tue, 26 Mar 2024 14:41:13 -0700 Subject: [PATCH 04/13] remove extra line --- docs/3-opt-edit-experiment.rst | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/3-opt-edit-experiment.rst b/docs/3-opt-edit-experiment.rst index 155bfd6a8..f7431c2e6 100644 --- a/docs/3-opt-edit-experiment.rst +++ b/docs/3-opt-edit-experiment.rst @@ -72,7 +72,6 @@ setup step:: ./benchpark setup benchmark/programmingmodel system --modifier=caliper - After the experiments in the workspace have completed running, a ``.cali`` file is created which contains the time metrics: From dd46e8005eeef1c7f8aa1b5f3103c2e2d771e56a Mon Sep 17 00:00:00 2001 From: Stephanie Brink Date: Tue, 26 Mar 2024 14:42:12 -0700 Subject: [PATCH 05/13] add following --- docs/3-opt-edit-experiment.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/3-opt-edit-experiment.rst b/docs/3-opt-edit-experiment.rst index f7431c2e6..e28dc0001 100644 --- a/docs/3-opt-edit-experiment.rst +++ b/docs/3-opt-edit-experiment.rst @@ -73,7 +73,7 @@ setup step:: ./benchpark setup benchmark/programmingmodel system --modifier=caliper After the experiments in the workspace have completed running, a ``.cali`` file -is created which contains the time metrics: +is created which contains the following time metrics: * Min time/rank: Minimum time (in seconds) across all ranks * Max time/rank: Maximum time (in seconds) across all ranks From 83ed46f7fde751eaebface7d6e9dcea916e43ab9 Mon Sep 17 00:00:00 2001 From: Stephanie Brink Date: Thu, 4 Apr 2024 11:28:39 -0700 Subject: [PATCH 06/13] reconfigure, separate modifier and caliper modifier sections - add table for different caliper modifiers --- docs/3-opt-edit-experiment.rst | 48 ++++++++++++++++++++++++---------- 1 file changed, 34 insertions(+), 14 deletions(-) diff --git a/docs/3-opt-edit-experiment.rst b/docs/3-opt-edit-experiment.rst index e28dc0001..df5c7ad5b 100644 --- a/docs/3-opt-edit-experiment.rst +++ b/docs/3-opt-edit-experiment.rst @@ -55,17 +55,24 @@ you can do so by upstreaming changes to Spack and/or Ramble, or working on your benchmark specification in ``benchpark/repo/${BENCHMARK}`` (see :doc:`add-a-benchmark` for details). -Applying the Caliper modifier ------------------------------ -In Benchpark, a modifier follows `Ramble Modifier +Modifiers +--------- +In Benchpark, a ``modifier`` follows the `Ramble Modifier `_ and is an abstract object that can be applied to a large set of reproducible -specifications, such as injecting performance analysis. In Benchpark, we have -implemented a Caliper modifier to build the specifications with the -instrumentation and profiling library, Caliper, in a platform-independent -manner. With the modifier enabled, Caliper is configured to use the SPOT -built-in configuration, which records a time profile. More documentation -on Caliper can be found `here `_. +specifications. Modifiers are intended to encasulate reusable patterns that +perform a specific configuration of an experiment. This may include injecting +performance analysis or setting up system resources. + +Applying the Caliper modifier +----------------------------- +We have implemented a Caliper modifier to build the specifications with the +instrumentation and profiling library in a platform-independent +manner. More documentation on Caliper can be found `here +`_. + +With the modifier enabled, Caliper is configured to one of the specified +configurations as shown in the table below. To turn on the Caliper modifier, add ``--modifier=caliper`` to the Benchpark setup step:: @@ -73,9 +80,22 @@ setup step:: ./benchpark setup benchmark/programmingmodel system --modifier=caliper After the experiments in the workspace have completed running, a ``.cali`` file -is created which contains the following time metrics: +is created which contains the collected performance metrics. + + +.. list-table:: Available caliper modifiers + :widths: 20 20 50 + :header-rows: 1 -* Min time/rank: Minimum time (in seconds) across all ranks -* Max time/rank: Maximum time (in seconds) across all ranks -* Avg time/rank: Average time (in seconds) across all ranks -* Total time: Aggregated time (in seconds) over all ranks + * - Caliper modifier + - Where applicable + - Metrics collected + * - Base + - Platform-independent + - Min time/rank: Minimum time (in seconds) across all ranks, Max time/rank: Maximum time (in seconds) across all ranks, Avg time/rank: Average time (in seconds) across all ranks, Total time: Aggregated time (in seconds) over all ranks + * - Top-down + - x86 CPUs + - Retiring, Bad speculation, Front end bound, Back end bound + * - CUDA + - CUDA GPUs + - time.gpu From 48e1d65aaf5d80e0d790d1ec0b548da61938d6d3 Mon Sep 17 00:00:00 2001 From: Stephanie Brink Date: Thu, 4 Apr 2024 11:35:31 -0700 Subject: [PATCH 07/13] multi-line list in table --- docs/3-opt-edit-experiment.rst | 18 ++++++++++++------ 1 file changed, 12 insertions(+), 6 deletions(-) diff --git a/docs/3-opt-edit-experiment.rst b/docs/3-opt-edit-experiment.rst index df5c7ad5b..9bdccc68f 100644 --- a/docs/3-opt-edit-experiment.rst +++ b/docs/3-opt-edit-experiment.rst @@ -87,15 +87,21 @@ is created which contains the collected performance metrics. :widths: 20 20 50 :header-rows: 1 - * - Caliper modifier - - Where applicable - - Metrics collected + * - Caliper Modifier + - Where Applicable + - Metrics Collected * - Base - Platform-independent - - Min time/rank: Minimum time (in seconds) across all ranks, Max time/rank: Maximum time (in seconds) across all ranks, Avg time/rank: Average time (in seconds) across all ranks, Total time: Aggregated time (in seconds) over all ranks + - | - Min time/rank: Minimum time (in seconds) across all ranks + | - Max time/rank: Maximum time (in seconds) across all ranks + | - Avg time/rank: Average time (in seconds) across all ranks + | - Total time: Aggregated time (in seconds) over all ranks * - Top-down - x86 CPUs - - Retiring, Bad speculation, Front end bound, Back end bound + - | - Retiring + | - Bad speculation + | - Front end bound + | - Back end bound * - CUDA - CUDA GPUs - - time.gpu + - | - time.gpu From 63f04ec8bfc9d38bfbd23ee7b6cf1c4249451b07 Mon Sep 17 00:00:00 2001 From: Stephanie Brink Date: Thu, 4 Apr 2024 22:48:02 -0700 Subject: [PATCH 08/13] update table --- docs/3-opt-edit-experiment.rst | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/docs/3-opt-edit-experiment.rst b/docs/3-opt-edit-experiment.rst index 9bdccc68f..53d91768f 100644 --- a/docs/3-opt-edit-experiment.rst +++ b/docs/3-opt-edit-experiment.rst @@ -74,15 +74,17 @@ manner. More documentation on Caliper can be found `here With the modifier enabled, Caliper is configured to one of the specified configurations as shown in the table below. -To turn on the Caliper modifier, add ``--modifier=caliper`` to the Benchpark +To turn on the Caliper modifier, add ``--modifier=`` to the Benchpark setup step:: - ./benchpark setup benchmark/programmingmodel system --modifier=caliper + ./benchpark setup benchmark/programmingmodel system --modifier= + +Valid values for ```` are found in the **Caliper Modifier** +column of the table below. After the experiments in the workspace have completed running, a ``.cali`` file is created which contains the collected performance metrics. - .. list-table:: Available caliper modifiers :widths: 20 20 50 :header-rows: 1 @@ -90,18 +92,18 @@ is created which contains the collected performance metrics. * - Caliper Modifier - Where Applicable - Metrics Collected - * - Base + * - caliper - Platform-independent - | - Min time/rank: Minimum time (in seconds) across all ranks | - Max time/rank: Maximum time (in seconds) across all ranks | - Avg time/rank: Average time (in seconds) across all ranks | - Total time: Aggregated time (in seconds) over all ranks - * - Top-down + * - caliper-topdown - x86 CPUs - | - Retiring | - Bad speculation | - Front end bound | - Back end bound - * - CUDA + * - caliper-cuda - CUDA GPUs - | - time.gpu From 2c75d078c6e5bcbb0ff15cad05d0d1a055c206ac Mon Sep 17 00:00:00 2001 From: pearce8 Date: Fri, 5 Apr 2024 07:46:28 -0500 Subject: [PATCH 09/13] Update 3-opt-edit-experiment.rst --- docs/3-opt-edit-experiment.rst | 24 ++++++++++-------------- 1 file changed, 10 insertions(+), 14 deletions(-) diff --git a/docs/3-opt-edit-experiment.rst b/docs/3-opt-edit-experiment.rst index 53d91768f..59b4dd456 100644 --- a/docs/3-opt-edit-experiment.rst +++ b/docs/3-opt-edit-experiment.rst @@ -31,7 +31,7 @@ Benchpark configuration files are organized as follows:: │ └── package.py └── repo.yaml -You can edit any of them to change the behavior of your experiments. +You can edit these configuration files to change the behavior of your experiments. System specification -------------------- @@ -66,23 +66,19 @@ performance analysis or setting up system resources. Applying the Caliper modifier ----------------------------- -We have implemented a Caliper modifier to build the specifications with the -instrumentation and profiling library in a platform-independent -manner. More documentation on Caliper can be found `here +We have implemented a Caliper modifier to enable profiling of Caliper-instrumented +benchmarks in Benchpark. More documentation on Caliper can be found `here `_. -With the modifier enabled, Caliper is configured to one of the specified -configurations as shown in the table below. - -To turn on the Caliper modifier, add ``--modifier=`` to the Benchpark +To turn on profiling with Caliper, add ``--modifier=`` to the Benchpark setup step:: ./benchpark setup benchmark/programmingmodel system --modifier= Valid values for ```` are found in the **Caliper Modifier** -column of the table below. - -After the experiments in the workspace have completed running, a ``.cali`` file +column of the table below. Benchpark will link the experiment to Caliper, +and inject appropriate Caliper configuration at runtime. After the experiments +in the workspace have completed running, a ``.cali`` file is created which contains the collected performance metrics. .. list-table:: Available caliper modifiers @@ -99,11 +95,11 @@ is created which contains the collected performance metrics. | - Avg time/rank: Average time (in seconds) across all ranks | - Total time: Aggregated time (in seconds) over all ranks * - caliper-topdown - - x86 CPUs + - x86 Intel CPUs - | - Retiring | - Bad speculation | - Front end bound | - Back end bound * - caliper-cuda - - CUDA GPUs - - | - time.gpu + - NVIDIA GPUs + - | - CUDA API functions (e.g., time.gpu) From 4d4483a2852834410c9106223d6235e7022c844d Mon Sep 17 00:00:00 2001 From: "Olga T. Pearce" Date: Tue, 16 Apr 2024 11:35:49 -0500 Subject: [PATCH 10/13] Adding modifiers to the help menu --- bin/benchpark | 59 +++++++++++++++++++++++++++++++++++++-------------- 1 file changed, 43 insertions(+), 16 deletions(-) diff --git a/bin/benchpark b/bin/benchpark index a5359a2e9..11b7fbf91 100755 --- a/bin/benchpark +++ b/bin/benchpark @@ -73,7 +73,7 @@ def source_location(): def benchpark_list(subparsers, actions_dict): list_parser = subparsers.add_parser( - "list", help="List available benchmarks and systems" + "list", help="List available experiments, systems, and modifiers" ) list_parser.add_argument("sublist", nargs="?") actions_dict["list"] = benchpark_list_handler @@ -106,6 +106,14 @@ def benchpark_systems(): return systems +def benchpark_modifiers(): + source_dir = source_location() + modifiers = [] + for x in os.listdir(source_dir / "modifiers"): + modifiers.append(x) + return modifiers + + def benchpark_get_tags(): f = source_location() / "tags.yaml" tags = [] @@ -131,30 +139,36 @@ def benchpark_get_tags(): def benchpark_list_handler(args): source_dir = source_location() sublist = args.sublist - benchmarks = benchpark_benchmarks() + experiments = benchpark_experiments() systems = benchpark_systems() + modifiers = benchpark_modifiers() if sublist == None: - print("Benchmarks/ProgrammingModel:") - for benchmark in benchmarks: - print(f"\t{benchmark}") + print("experiments:") + for experiment in experiments: + print(f"\t{experiment}") print("Systems:") for system in systems: print(f"\t{system}") else: - if sublist == "benchmarks": - print("Benchmarks:") - for benchmark in benchmarks: - print(f"\t{benchmark}") + if sublist == "experiments": + print("Experiments:") + for experiment in experiments: + print(f"\t{experiment}") else: if sublist == "systems": print("Systems:") for system in systems: print(f"\t{system}") else: - raise ValueError( - f'Invalid benchpark list "{sublist}" - must choose [benchmarks], [systems], or leave empty' - ) + if sublist == "modifiers": + print("Modifiers:") + for modifier in modifiers: + print(f"\t{modifier}") + else: + raise ValueError( + f'Invalid benchpark list "{sublist}" - must choose [experiments], [systems], [modifiers] or leave empty' + ) def benchpark_check_benchmark(arg_str): @@ -172,7 +186,7 @@ def benchpark_check_experiment(arg_str): experiments = benchpark_experiments() found = arg_str in experiments if not found: - out_str = f'Invalid benchmark/experiment "{arg_str}" - must choose one of: ' + out_str = f'Invalid experiment "{arg_str}" - must choose one of: ' for experiment in experiments: out_str += f"\n\t{experiment}" raise ValueError(out_str) @@ -201,6 +215,17 @@ def benchpark_check_tag(arg_str): return found +def benchpark_check_modifier(arg_str): + modifiers = benchpark_modifiers() + found = arg_str in modifiers + if not found: + out_str = f'Invalid modifier "{arg_str}" - must choose one of: ' + for modifier in modifiers: + out_str += f"\n\t{modifier}" + raise ValueError(out_str) + return found + + def benchpark_setup(subparsers, actions_dict): create_parser = subparsers.add_parser( "setup", help="Set up an experiment and prepare it to build/run" @@ -329,10 +354,12 @@ def benchpark_setup_handler(args): valid_benchmark = benchpark_check_benchmark(benchmark) debug_print(f"specified system = {system}") valid_system = benchpark_check_system(system) - if not (valid_benchmark and valid_system): + debug_print(f"specified modifier = {modifier}") + valid_modifier = benchpark_check_modifier(modifier) + if not (valid_benchmark and valid_system and valid_modifier): raise ValueError( - "Invalid benchmark/experiment and system provided: {0} {1}".format( - benchmark, system + "Invalid experiment, system, or modifier provided: {0} {1} {2}".format( + benchmark, system, modifier ) ) From 0386501fc27b1e14c35fbdc482641a9be14b342d Mon Sep 17 00:00:00 2001 From: "Olga T. Pearce" Date: Tue, 16 Apr 2024 13:50:05 -0500 Subject: [PATCH 11/13] List benchmarks and experiments --- bin/benchpark | 68 ++++++++++++++++++++++++++++----------------------- 1 file changed, 37 insertions(+), 31 deletions(-) diff --git a/bin/benchpark b/bin/benchpark index 11b7fbf91..58b0f5624 100755 --- a/bin/benchpark +++ b/bin/benchpark @@ -84,8 +84,7 @@ def benchpark_benchmarks(): benchmarks = [] experiments_dir = source_dir / "experiments" for x in os.listdir(experiments_dir): - for y in os.listdir(experiments_dir / x): - benchmarks.append(f"{x}/{y}") + benchmarks.append(f"{x}") return benchmarks @@ -94,7 +93,8 @@ def benchpark_experiments(): experiments = [] experiments_dir = source_dir / "experiments" for x in os.listdir(experiments_dir): - experiments.append(f"{x}") + for y in os.listdir(experiments_dir / x): + experiments.append(f"{x}/{y}") return experiments @@ -139,43 +139,49 @@ def benchpark_get_tags(): def benchpark_list_handler(args): source_dir = source_location() sublist = args.sublist + benchmarks = benchpark_benchmarks() experiments = benchpark_experiments() systems = benchpark_systems() modifiers = benchpark_modifiers() if sublist == None: - print("experiments:") + print("Experiments:") for experiment in experiments: print(f"\t{experiment}") print("Systems:") for system in systems: print(f"\t{system}") else: - if sublist == "experiments": - print("Experiments:") - for experiment in experiments: - print(f"\t{experiment}") + if sublist == "benchmarks": + print("Benchmarks:") + for benchmark in benchmarks: + print(f"\t{benchmark}") else: - if sublist == "systems": - print("Systems:") - for system in systems: - print(f"\t{system}") + if sublist == "experiments": + print("Experiments:") + for experiment in experiments: + print(f"\t{experiment}") else: - if sublist == "modifiers": - print("Modifiers:") - for modifier in modifiers: - print(f"\t{modifier}") + if sublist == "systems": + print("Systems:") + for system in systems: + print(f"\t{system}") else: - raise ValueError( - f'Invalid benchpark list "{sublist}" - must choose [experiments], [systems], [modifiers] or leave empty' - ) + if sublist == "modifiers": + print("Modifiers:") + for modifier in modifiers: + print(f"\t{modifier}") + else: + raise ValueError( + f'Invalid benchpark list "{sublist}" - must choose [experiments], [systems], [modifiers] or leave empty' + ) def benchpark_check_benchmark(arg_str): benchmarks = benchpark_benchmarks() found = arg_str in benchmarks if not found: - out_str = f'Invalid benchmark/experiment "{arg_str}" - must choose one of: ' + out_str = f'Invalid benchmark "{arg_str}" - must choose one of: ' for benchmark in benchmarks: out_str += f"\n\t{benchmark}" raise ValueError(out_str) @@ -186,7 +192,7 @@ def benchpark_check_experiment(arg_str): experiments = benchpark_experiments() found = arg_str in experiments if not found: - out_str = f'Invalid experiment "{arg_str}" - must choose one of: ' + out_str = f'Invalid experiment (benchmark/ProgrammingModel) "{arg_str}" - must choose one of: ' for experiment in experiments: out_str += f"\n\t{experiment}" raise ValueError(out_str) @@ -232,7 +238,7 @@ def benchpark_setup(subparsers, actions_dict): ) create_parser.add_argument( - "benchmark", type=str, help="The experiment (benchmark/ProgrammingModel) to run" + "experiment", type=str, help="The experiment (benchmark/ProgrammingModel) to run" ) create_parser.add_argument( "system", type=str, help="The system on which to run the experiment" @@ -336,34 +342,34 @@ def benchpark_setup_handler(args): experiments_root/ spack/ ramble/ - / + / / workspace/ configs/ (everything from source/configs/) - (everything from source/experiments/) + (everything from source/experiments/) """ - benchmark = args.benchmark + experiment = args.experiment system = args.system experiments_root = pathlib.Path(os.path.abspath(args.experiments_root)) modifier = args.modifier source_dir = source_location() debug_print(f"source_dir = {source_dir}") - debug_print(f"specified benchmark/ProgrammingModel = {benchmark}") - valid_benchmark = benchpark_check_benchmark(benchmark) + debug_print(f"specified experiment (benchmark/ProgrammingModel) = {experiment}") + valid_experiment = benchpark_check_experiment(experiment) debug_print(f"specified system = {system}") valid_system = benchpark_check_system(system) debug_print(f"specified modifier = {modifier}") valid_modifier = benchpark_check_modifier(modifier) - if not (valid_benchmark and valid_system and valid_modifier): + if not (valid_experiment and valid_system and valid_modifier): raise ValueError( "Invalid experiment, system, or modifier provided: {0} {1} {2}".format( - benchmark, system, modifier + experiment, system, modifier ) ) - workspace_dir = experiments_root / str(benchmark) / str(system) + workspace_dir = experiments_root / str(experiment) / str(system) if workspace_dir.exists(): if workspace_dir.is_dir(): @@ -387,7 +393,7 @@ def benchpark_setup_handler(args): print(f"Setting up configs for Ramble workspace {ramble_configs_dir}") configs_src_dir = source_dir / "configs" / str(system) - experiment_src_dir = source_dir / "experiments" / benchmark + experiment_src_dir = source_dir / "experiments" / experiment modifier_config_dir = source_dir / "modifiers" / modifier / "configs" ramble_configs_dir.mkdir(parents=True) ramble_logs_dir.mkdir(parents=True) From 40cc5a881379ee3596d8d5e068ce52787da6c644 Mon Sep 17 00:00:00 2001 From: "Olga T. Pearce" Date: Tue, 16 Apr 2024 15:19:35 -0500 Subject: [PATCH 12/13] Lint --- bin/benchpark | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/bin/benchpark b/bin/benchpark index 58b0f5624..ad24ba9da 100755 --- a/bin/benchpark +++ b/bin/benchpark @@ -238,7 +238,9 @@ def benchpark_setup(subparsers, actions_dict): ) create_parser.add_argument( - "experiment", type=str, help="The experiment (benchmark/ProgrammingModel) to run" + "experiment", + type=str, + help="The experiment (benchmark/ProgrammingModel) to run", ) create_parser.add_argument( "system", type=str, help="The system on which to run the experiment" From cb64772ee23a7d0841a78ed7b546b6af1a092ec5 Mon Sep 17 00:00:00 2001 From: "Olga T. Pearce" Date: Wed, 24 Apr 2024 12:35:31 -0700 Subject: [PATCH 13/13] Flattening list options, removing redundant error output --- bin/benchpark | 48 +++++++++++++++++++----------------------------- 1 file changed, 19 insertions(+), 29 deletions(-) diff --git a/bin/benchpark b/bin/benchpark index ad24ba9da..b900b4cb7 100755 --- a/bin/benchpark +++ b/bin/benchpark @@ -151,30 +151,26 @@ def benchpark_list_handler(args): print("Systems:") for system in systems: print(f"\t{system}") + elif sublist == "benchmarks": + print("Benchmarks:") + for benchmark in benchmarks: + print(f"\t{benchmark}") + elif sublist == "experiments": + print("Experiments:") + for experiment in experiments: + print(f"\t{experiment}") + elif sublist == "systems": + print("Systems:") + for system in systems: + print(f"\t{system}") + elif sublist == "modifiers": + print("Modifiers:") + for modifier in modifiers: + print(f"\t{modifier}") else: - if sublist == "benchmarks": - print("Benchmarks:") - for benchmark in benchmarks: - print(f"\t{benchmark}") - else: - if sublist == "experiments": - print("Experiments:") - for experiment in experiments: - print(f"\t{experiment}") - else: - if sublist == "systems": - print("Systems:") - for system in systems: - print(f"\t{system}") - else: - if sublist == "modifiers": - print("Modifiers:") - for modifier in modifiers: - print(f"\t{modifier}") - else: - raise ValueError( - f'Invalid benchpark list "{sublist}" - must choose [experiments], [systems], [modifiers] or leave empty' - ) + raise ValueError( + f'Invalid benchpark list "{sublist}" - must choose [experiments], [systems], [modifiers] or leave empty' + ) def benchpark_check_benchmark(arg_str): @@ -364,12 +360,6 @@ def benchpark_setup_handler(args): valid_system = benchpark_check_system(system) debug_print(f"specified modifier = {modifier}") valid_modifier = benchpark_check_modifier(modifier) - if not (valid_experiment and valid_system and valid_modifier): - raise ValueError( - "Invalid experiment, system, or modifier provided: {0} {1} {2}".format( - experiment, system, modifier - ) - ) workspace_dir = experiments_root / str(experiment) / str(system)