v0.2.1

This is a bugfix release that contains a number of small fixes in transformations and Scheduler.

What's New

• Utility methods have been added to CallStatement, which simplify inspecting, validating and converting keyword-arguments to positional arguments (see #235)
• The batch-processing module loki.bulk has been renamed to loki.batch

What's Changed

• kwargs utilities by @MichaelSt98 in #235
• Allow to ignore specific dimensions in "shift to zero indexing" by @MichaelSt98 in #236
• Add 'reverse_traversal=True' to DerivedTypeArgumentsTransformation manifest by @MichaelSt98 in #238
• Create a pid-specific temporary directory and clean it up at the end by @reuterbal in #261
• SCC-HOIST: Hoist variables as kwargs (optionally) by @MichaelSt98 in #237
• GlobalVarHoistTransformation: fix for functions/inline calls by @MichaelSt98 in #240
• Support colon notation for all dimensions in flatten_arrays by @MichaelSt98 in #239
• Small CMake layer fixes for SL by @awnawab in #248
• Rename bulk->batch and create ir sub-package by @mlange05 in #258
• SingleColumn: Demote arrays that are not used at all in the body by @mlange05 in #259
• Scheduler: Fix handling of external module procedures by @reuterbal in #263

Full Changelog: v0.2.0...v0.2.1

v0.2.0

This release contains a rewrite of Loki's Scheduler, which is responsible for planning and executing batch transformations across complex source trees. Compared to the original implementation, it is more flexible, enables handling of more dependency types (data dependencies, type dependencies as well as control flow dependencies) and is faster. Additional capabilities of pruning the dependency graph have been added as part of this, and the new discovery mechanism may make changes to Scheduler config files necessary. See the expanded documentation section for more details.

No other changes are included in this release and in case of problems we advise to report them as issues and stay on v0.1.7 in the meantime.

What's Changed

• The new Scheduler by @reuterbal in #213
• Update Github actions to get rid of node.js warnings by @reuterbal in #257

Full Changelog: v0.1.7...v0.2.0

v0.1.7

This is the final release of the v0.1 version of Loki. The new v0.2 will become available soon and use a rewritten Scheduler implementation for batch processing.

Most changes in this release are bugfixes, minor improvements or preparatory work for the new Scheduler integration. See below for the full list.

What's new

• The file parsing speed has been improved, which should make the processing of large source trees significantly faster (#229, #241, #242, #245).
• A new set of coding standards checks has been added, corresponding to the new IFS Arpège coding standards. This includes only three rules currently but will be expanded going forward (#247).
• The pool allocator transformation inserts the argument for the scratch space now as integer variables instead of a dedicated derived type. This was found to avoid an allocation on NVIDIA GPUs and yield performance improvements (#214).

What's Changed

• Create separate ModuleWrapTransformation from DependencyTransformation by @reuterbal in #197
• Transformation configuration and SchedulerConfig update by @mlange05 in #191
• Pragma-driven subroutine inlining and associated utilities by @mlange05 in #198
• introduce 'flatten_arrays()' (to overcome pointer hack) by @MichaelSt98 in #199
• Array shadowing bugfix for inliner by @skarppinen in #202
• Fix for SCC-HOIST, regarding wrong (hoisted) argument(s) (indexing) i… by @MichaelSt98 in #206
• Refactored Global Variable Offload by @reuterbal in #207
• Fixes to minor issues related to SCC HOIST by @skarppinen in #211
• introduce flag to allow removing all derived types by @MichaelSt98 in #215
• CMake: Fix DERIVE_ARGUMENT_SHAPE_ARRAY and argument handling by @mlange05 in #217
• Recursive inlining via InlineTransform and associated fixes by @mlange05 in #205
• Minor fixes to frontends and IR nodes by @reuterbal in #212
• Stack/Pool Allocator: pass stack as integer(s) by @MichaelSt98 in #214
• Frontend: Fix bug for multi-line pragmas and add short test by @mlange05 in #221
• assumed size array handling for 'normalize_array_shape_and_access' by @MichaelSt98 in #218
• Nested derived type bug fix by @rolfhm in #192
• C/C++ De/Reference by @MichaelSt98 in #223
• allow F2C transpilation using c_ptr (switch for old behaviour or new … by @MichaelSt98 in #219
• Github actions: Only load SSH_KEY from secrets if available by @reuterbal in #228
• f2c transpile via convert by @MichaelSt98 in #208
• Global var hoisting by @MichaelSt98 in #226
• Allow for limiting resolve_sequence_association to procedures that are inlined in loki-transform convert by @skarppinen in #225
• Performance optimisations for frontend parsing/sanitising by @mlange05 in #229
• Enabling SCC-Stack for EC-physics, part 2 by @mlange05 in #222
• Frontend Transformer optimisation by @mlange05 in #241
• FParser: Perform in-place scope attachment during parse by @mlange05 in #242
• Subroutine: Only clone symbol when inferring from allocatable by @mlange05 in #245
• Loki-lint: First 3 rules for new IFS-Arpégé coding standards by @reuterbal in #247

Full Changelog: v0.1.6...v0.1.7

v0.1.6

This release primarily contains bugfix and maintenance changes and few new features. It is intended as a stable basis before a set of breaking changes will be made for the next release. These will primarily relate to Scheduler behaviour and a new, consolidated config file format.

What's new

• A new utility by @rolfhm allows to resolve sequence association (#173)
• The disable property in the Scheduler config allows now wildcards/simple patterns (#194)
• A new utility by @skarppinen allows to extract internal subprograms from procedures to convert them into standalone procedures (#181)
• Transformation classes have now static properties that define how the scheduler should traverse the dependency graph, e.g., forward/reverse traversal, recursion into contained scopes, or file graph traversal (#154)

The full list of changes:

• Fix vector section trimming in driver loop by @awnawab in #169
• TransformInline: Fix rescoping in expression substitution by @mlange05 in #170
• Fix deep-cloning of subroutiens and modules (fix #174) by @mlange05 in #175
• Sourcefile does not have a filepath by @joscao in #177
• InlineMember: rename duplicate locals in the body by @rolfhm in #172
• Extract/Improve Polyhedron class by @joscao in #178
• Provide linear algebra utility by @joscao in #179
• add 'kernel_only=True' to RemoveCallsTransformation by @MichaelSt98 in #185
• Display dataflow analysis (if attached) in IR graph by @joscao in #183
• Fix handling of empty files in frontends (fix #186) by @reuterbal in #187
• Transformation utility to fix sequence association by @rolfhm in #173
• Minor transformation fixes by @reuterbal in #188
• Small parsing fixes by @awnawab in #190
• pool_allocator handles range indices by @rolfhm in #193
• Generic enrichment process by @reuterbal in #189
• Allow wildcards in disable list for scheduler by @mlange05 in #194
• Utility transformation for creating standalone subroutines from contained subroutines by @skarppinen in #181
• Static Transformation properties (manifest) by @mlange05 in #154

New Contributors

• @skarppinen made their first contribution in #181

Full Changelog: v0.1.5...v0.1.6

v0.1.5

Next to a large number of small bug fixes and improvements, this release introduces a number of API and behaviour changes, that have the potential to break existing workflows using Loki:

What's changed

• Expression nodes no longer have a source property (#118)
• The monolithic SCC wrapper transformation has been replaced by individual, composable transformation steps (#124)
• Statement function injection is now happening on-the-fly in the parse tree translation to Loki-IR (#132)
• API under ExpressionMapper has been changed, making behaviour static class properties (#138)
• The ecphys entry point of loki-transform.py has been retired and fully integrated into the convert command (#142)
• Transformation no longer recurses by default into contained scopes (#136)
• The CMake integration has been refactored, consolidating on a single CMake function loki_transform to call the loki-transform.py script, and the loki_transform_target function to automatically update a target's source files and copy compilation attributes for transformed files. loki_transform_target is retained for compatibility but marked as deprecated and will be removed in the future (#166)

What's new

• A transformation that allows inlining contained member subroutines (#130)
• The hoisting mechanism in the SCC transformations has been replaced by a recursive hoisting transformation (#156)
• SCC transformations can now also be applied to loops in the driver routine (#150)
• A graphical visualization of the Loki IR using Graphviz via the pretty_visualize utility function (#146)

The full list of changes

• Hotfix: Upload documentation to Sites for main branch by @reuterbal in #127
• Remove ExprMetadataMixin by @awnawab in #118
• Remove SCC wrapper transformation by @mlange05 in #124
• Skip tests if graphviz is not installed or functional by @rolfhm in #129
• SCC: Better logging and warning for "bad" code constructs by @mlange05 in #125
• Linter debug rule fixes by @awnawab in #126
• Make derived argument flattening optional in loki_transform convert by @mlange05 in #139
• Statement function parsing on-the-fly by @mlange05 in #132
• Fix type update behaviour in expression clones by @mlange05 in #138
• Expression mappers: Remove recurse_to_parent option and recurse by default by @reuterbal in #140
• Fix for enrich calls bug by @awnawab in #141
• Remove "ecphys" entry point from Loki_transform.py by @mlange05 in #142
• Single declarations for hoisted variables in recursive hoist transformation (fixes #143) by @reuterbal in #144
• Disable default recursion for Transformation.apply by @reuterbal in #136
• fix for argument shape regarding module imports by @MichaelSt98 in #148
• Provide a graphical representation of the IR using graphviz by @joscao in #146
• Add DEVICEPTR annotations to data region in driver loop by @awnawab in #145
• Fix cudafor import in driver for SCC CUF by @reuterbal in #157
• Make DataflowAnalysisAttacher/Detacher true in-place transform by @awnawab in #158
• Member inlining utility transform by @mlange05 in #130
• Bugfix: Memory error in Statement functions test by @reuterbal in #159
• Minor fix, k is undeclared by @rommelDB in #160
• Loki-SCC: Adapt to use recursive hoisting transformation by @mlange05 in #156
• CMake refactoring for loki_transform by @reuterbal in #166
• Loki_transform: Re-instate argument shape derivation in convert by @mlange05 in #149
• Extend SCC transformation to loops in driver by @MichaelSt98 in #150
• Save attribute (fix #164) by @reuterbal in #165
• bugfix : symbols from the inlined member's dummies were hoisted in the calling routine by @JoeffreyLegaux in #162
• Call remove call and remove hook transformations on contained subroutines by @rolfhm in #163
• Update version number to 0.1.5 by @reuterbal in #168

New Contributors

• @joscao made their first contribution in #146
• @rommelDB made their first contribution in #160
• @JoeffreyLegaux made their first contribution in #162

Full Changelog: v0.1.4...v0.1.5

v0.1.4

What's new:

• The SCC transformation has been split into multiple individual transformations (see #81). The current SCC transformation is now only a wrapper around these individual transformations and will be retired in the next release.
• A rewrite of the DerivedTypeArgumentsTransformation that can now handle flattening of derived types across multiple call layers and for nested derived types (#51)
• A debugging linter rule aimed at identifying certain anti-patterns and problems in source code (#116)
• Moved to pyproject.toml based package installation (#110)
• Added ecWAM regression testing (#119)

What's been fixed:

• Pydantic version number has been fixed <2.0 (#109)
• Various fixes the to the CI testing infrastructure
• Compatibility with Python 3.11 (#107)

The full list of changes:

• SCC modularization by @awnawab in #81
• FGen: Fix bug for inline-conditionals with multi-line output by @mlange05 in #97
• Pool allocator update by @awnawab in #92
• Various bugfixes by @reuterbal in #93
• Scheduler: Only apply transformations to requested subroutines by @mlange05 in #100
• Rewrite of DerivedTypeArgumentsTransformation by @reuterbal in #51
• SCC updates needed for ecWam by @awnawab in #95
• SCC: Insert vector_length directive into the gang loop by @reuterbal in #98
• Loki: Add a 1% code coverage degradation threshold by @mlange05 in #112
• Depend on pydantic<2.0 by @reuterbal in #109
• Python 3.11 compatibility by @reuterbal in #107
• Codecov: Remove old config, drop threashold, disable patch checking by @mlange05 in #114
• ECPhys: Add a "kernel-only" option to remove calls by @mlange05 in #111
• Migrate to pyproject.toml package installation by @reuterbal in #110
• Fix directive comparison to literal (fix #113) by @reuterbal in #115
• Debugging linter rules by @awnawab in #116
• Fix #120: Test and fix constructor arg conflict in FindExpressionRoot by @reuterbal in #121
• Add ecWam regression tests by @awnawab in #119
• Expressions: Add test for containment behaviour of expressions by @mlange05 in #122

Full Changelog: v0.1.3...v0.1.4

v0.1.3

What's new:

• A TemporariesPoolAllocatorTransformation (see #64 and #90) that allocates a large scratch space in the driver and maps temporary arrays in offloaded compute kernels to this scratch space, thus eliminating the overhead of temporary allocations by the runtime. The size of the scratch space is automatically determined via an analysis pass.
• A GlobalVarOffloadTransformation (see #66) that automatically inserts !$acc declare create annotations for global module variables, and ensures they are updated in the driver.
• A TypeboundProcedureCallTransformation (see #54) that replaces calls to typebound procedures by a direct call to the relevant procedure, inserting the derived type argument as required.
• Extension of frozen dataclasses to scoped nodes in the Loki IR (see #86). This introduces stricter type checking on IR node arguments, thus preempting some common errors.

What's been fixed:

• A small API change was introduced with respect to Module.typedefs, which now returns the list of TypeDef nodes rather than a dict. The dict is available via Module.typedef_map, in line with the nomenclature for imports and variables.

The full list of changes:

• Loki-transform: Expose "directive" option through script and CMake by @mlange05 in #79
• Loki-transform: Expose directive argument to loki_transform_convert by @mlange05 in #82
• acc routine pragmas fix by @awnawab in #80
• Transformation to resolve calls to typebound procedures by @reuterbal in #54
• Support for multiple kernel calls in DataOffloadTransformation by @mlange05 in #83
• Frontend: Add sanitization of fypp annotations to REGEX frontend by @mlange05 in #67
• SCC: Test and small bug fixes for resolve_masked_statement by @mlange05 in #84
• SCC-hoist: Add test and fix corner case, where no arrays are hoisted by @mlange05 in #85
• IR: Make ScopedNode classes frozen datasets by @mlange05 in #86
• Homogenize access to typedefs and imports in program units by @reuterbal in #87
• Pool allocator transformation by @reuterbal in #64
• Include Stack transformation in CLOUDSC regression testing by @reuterbal in #90
• Pragma regions detach bug fix by @awnawab in #89
• Global variable offload transformation by @awnawab in #66
• Dataflow associate block fix by @awnawab in #91

Full Changelog: v0.1.2...v0.1.3

v0.1.2

This Loki release includes the following new features:

• Transformation to remove utility routine calls by @mlange05 in #45
• IR: Use frozen dataclasses to enforce immutablity - Part 1 by @mlange05 in #15
• Add inline function calls as scheduler dependencies by @awnawab in #49
• Scheduler enhancements for dealing with derived types by @reuterbal in #50

The following changes incur a variation of the previous default behaviour:

• Make rescoping in post_apply optional by @reuterbal in #56

What's Changed

• Fix installation by @reuterbal in #40
• Fix #36: regex frontend hangs for function declaration followed by subroutine by @reuterbal in #38
• Fix #37: regex frontend hangs if end is missing subroutine by @reuterbal in #39
• Fix #41: Correctly substitute identical nodes in tuple mapping by @reuterbal in #42
• Various Pygen-backend fixes by @mlange05 in #43
• Documentation fixes by @reuterbal in #53
• SCC transformation bug fixes by @awnawab in #47
• Linter: Internalize functionality and scheduler integration stub by @reuterbal in #44
• Dataflow analysis bug fixes by @awnawab in #48
• Dataflow analysis multi-conditional live_symbols bug by @awnawab in #70
• Fix multi-call hoisting in scc-hoist and extend test by @mlange05 in #71
• Limited recursion and made boz code gfortran compliant by @rolfhm in #65
• Procedure interface scheduler bug fix by @awnawab in #68
• Additional fixes for IR by @reuterbal in #55
• Fix documentation CI for PRs from forks and separate pytest and regression tests in Github actions by @reuterbal in #73
• loki-transform: Small CMake refactoring, C preprocessing, absolute paths by @reuterbal in #52
• Bug fix in dependency transform for routines declared using explicit interface by @awnawab in #72
• Remove unused import and variable from hook by @rolfhm in #61
• Install ant via APT for Github actions by @reuterbal in #75
• Subroutine: Fix array declarations in interface generation by @mlange05 in #60
• Add module imports as scheduler dependencies by @awnawab in #62
• Fix order of USE statements and IMPLICIT NONE by @awnawab in #74

New Contributors

• @awnawab made their first contribution in #47
• @rolfhm made their first contribution in #65

Full Changelog: v0.1.1...v0.1.2

v0.1.1

This Loki release includes the following new features:

• A CUDA Fortran (CUF) backend (#2)
• A generic hoist functionality for driver-side allocation of temporary arrays, now even across multiple call levels (#3, #13)
• Support for implied-do (#5)
• New ParametriseTransformation to replace runtime parameters with compile time constant (#10)
• New transformation utility single_variable_declaration (#12)
• New SCC-CUF transformation for CLOUDSC (#14)
• Scheduler parses source files in reverse topological order, thus implicitly enriching calls (#17, #22)
• New demote_variables transformation utility (#24)
• Support for subroutine suffixes (#18)
• On-demand debugging by automatically attaching debugger on excdeptions via env variable LOKI_DEBUG=1 (#18)

Many thanks to everyone who reported issues and helped improving Loki. Thanks to your contributions, we fixed a number of bugs, including:

• Raise better error messages for PROGRAM constructs (#5)
• Remove redundant declarations after applying inline_constant_parameters (#9)
• Fixed numpy version until f90wrap is compatible with new release (#11)
• Fixed source handling in OMNI frontend (#27)
• Fixed interface blocks in subroutines in the REGEX frontend (#30)
• Fixed formatting of long source code lines (#18)
• Fixed case-sensitivity issues in Scheduler (#32)
• Fixed infinite recursion issue in expression substitutions (#34)

v0.1.0

This is the first public release of Loki under an Apache-2 license.