Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Utility transformation for creating standalone subroutines from contained subroutines #181

Merged
merged 22 commits into from
Dec 5, 2023

Conversation

skarppinen
Copy link
Contributor

This PR adds a function named lift_contained_subroutines (for a lack of a better name) which processes the contained subroutines of a loki.Subroutine such that they are converted to "stand alone subroutines" that have no global dependencies. To do this:

  1. all global bindings from the point of view of the contained subroutine(s) are introduced as imports or dummy arguments to the contained subroutine(s).
  2. all calls to the contained subroutines in the parent are modified accordingly.

To understand the basic idea of what the function does, consider the next example.
The following subroutine (with a contained subroutine "inner" in the CONTAINS block)

subroutine outer()
    integer :: y
    integer :: o
    o = 0
    y = 1
    call inner(o)
    contains
    subroutine inner(o)
       integer, intent(inout) :: o
       integer :: x
       x = 4
       o = x + y ! Note, 'y' is "global" here!
    end subroutine inner
end subroutine outer

is transformed into a list of loki.Subroutines, where the transformed "parent" (i.e "outer") comes first:

subroutine outer()
    integer :: y
    integer :: o
    o = 0
    y = 1
    call inner(o, y) ! 'y' now passed as argument.
    contains
end subroutine outer

and the transformed children (i.e in this case only "inner") come next:

subroutine inner(o, y)
       integer, intent(inout) :: o
       integer, intent(inout) :: y
       integer :: x
       x = 4
       o = x + y ! Note, 'y' is no longer "global"
end subroutine inner

Naturally, multiple contained subroutines are supported as well.

Some remarks:

  • The current implementation always sets the intent of "resolved variables" (i.e 'y' in the above example) as "inout", unless the variable to be resolved has an explicitly specified intent in the parent routine ('outer').
  • If a variable is to be resolved that lacks a definition in the parent scope, an exception is thrown.
  • If a variable to be resolved is a member of a derived type (say 'a%b'), the whole derived type (i.e 'a') is introduced as a dummy argument to the contained subroutine.
  • If a variable to be resolved is defined via an import, the import is added to the contained subroutine.
  • If the definition of a variable to be resolved depends on imports or other variables, the necessary imports are added to the contained subroutine, and the variables are added as arguments (if need be). These cases occur for example, when the variable to be resolved (below 'x') is defined for example by:
USE parkind1, only: jprb, jpim
INTEGER(KIND=jpim) :: klon
REAL(KIND=jprb) :: x(klon) ! to resolve 'x', import 'jprb' and introduce 'klon' as argument.
  • resolve_associates is called for each contained subroutine.

The tests cover the above cases (excluding resolving associates) and some other things (see docstring for each test). As additional "field testing", I have tested this (as a replacement for inlining) in the ACRANEB2 dwarf, where as part of an SCC pipeline this transform handles all Fortran constructs found there and produces code that compiles and provides correct results on NVIDIA GPUs.

Copy link

codecov bot commented Nov 9, 2023

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (955a9da) 92.21% compared to head (fa1dd79) 92.24%.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #181      +/-   ##
==========================================
+ Coverage   92.21%   92.24%   +0.02%     
==========================================
  Files          93       94       +1     
  Lines       16839    16903      +64     
==========================================
+ Hits        15528    15592      +64     
  Misses       1311     1311              
Flag Coverage Δ
lint_rules 96.22% <ø> (ø)
loki 92.22% <100.00%> (+0.03%) ⬆️
transformations 91.44% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Collaborator

@reuterbal reuterbal left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First of all, thanks for this contribution and apologies for the long time I took to review this. This does look very useful and I'm keen to bring this in.

There are a few formal code checks that failed, such as copyright headers and pylint warnings, which I hope you could take care of.

I have also left a few comments already for things that I think could be improved. Generally, you have re-implemented frequently some functionality that is already provided via convenience API and I would encourage you to use that whenever possible, since it will make maintenance and compatibility of your utility easier in the long run by relying on concepts abstracted from IR implementation details. Also, by splitting the utility we would gain composable functionality and a cleaner control flow. I have marked this accordingly in the code.

Feel free to ask questions, otherwise please ping me once you think this is ready for another look, at which stage I'll also pay a bit more attention to the details of the symbol resolution of the lifted routine (which I have only skimmed at this stage).

Thanks again for this contribution, and as a final request: please add your name to AUTHORS.md as part of this PR ;-)

loki/transform/transform_lift_contained_subroutines.py Outdated Show resolved Hide resolved
tests/test_lift_contained_subroutines.py Outdated Show resolved Hide resolved
loki/transform/transform_lift_contained_subroutines.py Outdated Show resolved Hide resolved
loki/transform/transform_lift_contained_subroutines.py Outdated Show resolved Hide resolved
loki/transform/transform_lift_contained_subroutines.py Outdated Show resolved Hide resolved
loki/transform/transform_lift_contained_subroutines.py Outdated Show resolved Hide resolved
loki/transform/transform_lift_contained_subroutines.py Outdated Show resolved Hide resolved
loki/transform/transform_lift_contained_subroutines.py Outdated Show resolved Hide resolved
tests/test_lift_contained_subroutines.py Outdated Show resolved Hide resolved
loki/transform/transform_lift_contained_subroutines.py Outdated Show resolved Hide resolved
Copy link
Collaborator

@mlange05 mlange05 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apologies for taking so long to get back to this. The utility certainly looks very useful, and a great contribution to have in our housekeeping arsenal. As @reuterbal already pointed out, we'll need appropriate copyright headers and and some of the outer interfacing could be slightly improved, so that we can use this in the future in a more composable way with other transformations. The test coverage, however, seems very thorough which is really great to see! 😄

I also noted that the utility routine itself does many little things in that main loop, which could indicate that breaking it down further might make the corner-cases more manageable. I'd like to have another quick look once most of the main issues have been addressed (and I promise to be quicker this time 😉 ), so leaving this in "comment" state for now. But overall this is looking very nice!

return impo.clone()
raise Exception(f"variable '{varname}' was not found from the specified import objects.")

routine = routine.clone()
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with this point, and additionally note that the overall transformation now modifies the enclosing Module. I would therefore propose to wrap this utility in a Transformation, so that the above snippet can be applied to a Sourcefile/Module effectively in a full batch-processing scenario.

loki/transform/transform_lift_contained_subroutines.py Outdated Show resolved Hide resolved
@skarppinen
Copy link
Contributor Author

Thanks @reuterbal and @mlange05 for your valuable comments. Based on a quick look I think I agree with all of them. (many of them I could anticipate, but decided to post this early without too much polishing)

I am writing this message just to acknowledge that I've seen your comments. However, I am quite busy at the moment, and it may well be until next month that I have time to address them.

@skarppinen
Copy link
Contributor Author

Thanks again for the comments. I think I addressed most of them and this is looking a lot better already.
However, I think the symbol resolution could be improved, since it is currently based on using strings. Should I be using TypedSymbols or something like that?

I did not yet fulfil @mlange05's wish to make a Transformation out of this, but I think this is a good idea. However, I suggest a course of action to first fix the symbol resolution to make this more "Lokian", and after that, wrap everything into a Transformation that can handle Modules and Sourcefiles too.

…ariable objects, and scopes to do resolution. add a few new tests to cover cornercases missed before. remove resolveassociates, not required anymore.
@skarppinen
Copy link
Contributor Author

I made a reimplementation of the proposed functionality that I think is a lot better. Instead of referring to variables using strings, I now use the variable objects and scope information provided by Loki to do the symbol/variable resolution. I also removed resolve_associates as it seems it is not needed. Furthermore, I added a few new tests to handle cornercases I had overlooked before.

Copy link
Collaborator

@reuterbal reuterbal left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Many thanks for the changes, I agree that the implementation looks much better now. I very much like how you've structured the routine. It appears very logical to me and the control flow was easy to read and follow!

I have left a number of remarks, some only stylistic, others pointing out a few potential problems. Please have a look if you agree with them.

Canonical style recommendation (PEP-8) is also to have two empty lines between free function definitions.

Testing is also very extensive, which is great. To ensure this works also for slightly more complex dummy argument interfaces, would you mind adding something where the inner function and subroutine expect also by default some argument? This can be just an addition to an existing test, no need to add a new one. E.g.

function inner(i) result(z)
  integer, intent(in) :: i
  integer :: y
  integer :: z
  y = i
  z = x + y
end function inner

And then test with various uses, like

y1 = inner(1)
y2 = inner(i=1)
y3 = inner(y1)

Since we want to ensure that all transformation utilities work independently of the specific Loki frontend that has been used, would you mind adding test parameterisations for the frontend? See the in-line comment for this.

Wrapping this in a transformation would indeed be nice, but I think this one has already enough complexity. So, that should ideally come as a separate PR.

loki/transform/transform_extract_contained_procedures.py Outdated Show resolved Hide resolved
loki/transform/transform_extract_contained_procedures.py Outdated Show resolved Hide resolved
loki/transform/transform_extract_contained_procedures.py Outdated Show resolved Hide resolved
loki/transform/transform_extract_contained_procedures.py Outdated Show resolved Hide resolved
loki/transform/transform_extract_contained_procedures.py Outdated Show resolved Hide resolved
tests/test_extract_contained_procedures.py Outdated Show resolved Hide resolved
tests/test_extract_contained_procedures.py Outdated Show resolved Hide resolved
tests/test_extract_contained_procedures.py Outdated Show resolved Hide resolved
tests/test_extract_contained_procedures.py Outdated Show resolved Hide resolved
@skarppinen
Copy link
Contributor Author

Thanks for the comments again @reuterbal. The latest commit addresses them.

Copy link
Collaborator

@reuterbal reuterbal left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks so much for implementing the requested changes.
On the implementation side, I have only one more remark about style, which I would appreciate if we could follow to ensure consistent appearance.

Unfortunately, the other (more strict) Fortran frontends seem unhappy with some of the source code in the tests. OMNI refuses to parse incomplete code where it doesn't have the definition of an imported module. This is a known limitation and it is permissible to skip that frontend in these tests - I have pointed out which those are and how to mark this in the parameterisation.

Also, OMNI has found a genuine bug in the Fortran source for one of the tests.

FInally, one test fails with OFP at the end and I'm not clear what the cause is for that.

loki/transform/transform_extract_contained_procedures.py Outdated Show resolved Hide resolved
tests/test_extract_contained_procedures.py Outdated Show resolved Hide resolved
tests/test_extract_contained_procedures.py Outdated Show resolved Hide resolved
tests/test_extract_contained_procedures.py Outdated Show resolved Hide resolved
tests/test_extract_contained_procedures.py Outdated Show resolved Hide resolved
tests/test_extract_contained_procedures.py Outdated Show resolved Hide resolved
tests/test_extract_contained_procedures.py Outdated Show resolved Hide resolved
tests/test_extract_contained_procedures.py Outdated Show resolved Hide resolved
tests/test_extract_contained_procedures.py Show resolved Hide resolved
tests/test_extract_contained_procedures.py Outdated Show resolved Hide resolved
…ntend issue with ofp frontend, fix issues in 4 tests with implicit none appearing before module imports, added xfails for omni frontend for missing modules which are not relevant for features tested
@skarppinen
Copy link
Contributor Author

Thanks @reuterbal, I fixed the issues you pointed out.

Copy link
Collaborator

@reuterbal reuterbal left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you very much for taking care of the failing tests. There's one final bit missing, otherwise I think it is ready to be merged.

tests/test_extract_contained_procedures.py Outdated Show resolved Hide resolved
Copy link
Collaborator

@reuterbal reuterbal left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Many thanks for the patience and endurance getting this over the line. This looks great now!

@reuterbal reuterbal added the ready for merge This PR has been approved and is ready to be merged label Dec 5, 2023
@skarppinen
Copy link
Contributor Author

Thanks @reuterbal, this was a good learning experience.

Copy link
Collaborator

@mlange05 mlange05 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indeed, just wanted to add that this looks really great now, and the rigorous test coverage is much appreciated. I only skimmed the final version, but test look solid, so GTG from me too.

And also, another big thank you for the patient and rigorous PR and review process. The final result looks fantastic, and a great full first contribution. Many, many thanks @skarppinen and @reuterbal

@reuterbal reuterbal merged commit 4fa3651 into ecmwf-ifs:main Dec 5, 2023
8 of 12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ready for merge This PR has been approved and is ready to be merged
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants