Skip to content

Commit

Permalink
misc
Browse files Browse the repository at this point in the history
  • Loading branch information
@DavidSpickett
DavidSpickett committed Nov 14, 2024
1 parent 8b19099 commit 54df17f
Showing 1 changed file with 67 additions and 57 deletions.
124 changes: 67 additions & 57 deletions content/posts/2024-11-05-flang-new.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -9,13 +9,14 @@ LLVM has included a Fortran compiler "Flang" since LLVM 11. What you might not
know is that the name of Flang's binary was not `flang`, but `flang-new`.

The LLVM 20 release brings an end to the era of `flang-new`. The
community has decided it has reached a level of quality worthy of a new name.
community has decided Flang has reached a level of quality that is worthy of a
new name.

The "new" name? That's right, `flang`.
The "new" name? You guessed it, `flang`.

In this article I will explain why this simple change actually represents a
major milestone for the Flang project. Covering its almost 10 year journey to
landing in the LLVM Project and adoption of the latest LLVM frameworks like MLIR.
major milestone for the Flang project. Covering Flang's almost 10 year journey to
into the LLVM Project and its adoption of one of LLVM's hottest new features, MLIR.

If you want to try `flang` right now, you can download it now or try it
in your browser using Compiler Explorer.
Expand All @@ -33,7 +34,7 @@ kept the industry investing in it for the last 70 years.
IBM has an excellent [article](https://www.ibm.com/history/fortran) on the history
of Fortran and I encourage you to look at the
["Programmer's Primer for Fortran"](https://archive.computerhistory.org/resources/text/Fortran/102653987.05.01.acc.pdf)
as well.
if you want to know more.

The name Fortran came from "Formula Translation" and reading the Programmer's
Primer you will see why. A focus on the mathematical use case, freeing programmers
Expand Down Expand Up @@ -69,15 +70,16 @@ paraphrased here:
* Support Fortran tool development by being based on existing LLVM frameworks.
* Support Fortran language experimentation for future language standards proposals.

The last one can be explained by comparison to Clang. For proposals of new features
for C++, a common path is to implement them on top of either Clang
or GCC to prove they will work. However, the two compilers have different strengths.
Having both available allows us to find those problems early and so, we hope,
will having Flang and gfortran.
The last one can be explained by comparison to Clang. For C++ feature, a common
path is to implement them on top of Clang, GCC, or both, to prove the feature can work.
Implementing the feature multiple times in different compilers uncovers any
assumptions in the design that may be a problem for one compiler or another.

Finally, though Fortran is an ISO standard we know that standards can be ambiguous,
and implementations can make mistakes. Building a new compiler is a chance to
rethink those decisions and find corners of the standards that have been neglected.
In the same way, Flang and gfortran can provide that diversity.

Finally, though Fortran is an ISO standard, standards can be ambiguous
and implementations can make mistakes. A new compiler is a chance to
rethink those decisions and find parts of the standards that have been neglected.

Jeff Hammond (NVIDIA) had this to say on the topic:

Expand All @@ -102,7 +104,7 @@ so we will start there.
STMicroElectronics.
* **July 2013** PGI is [sold to NVIDIA](https://www.theregister.com/2013/07/30/nvidia_buys_the_portland_group/).

Next, we saw the first steps towards the first iteration of Flang.
In late 2015 we saw the first signs of what would become "Classic Flang".

* **November 2015** NVIDIA is part of the US Department of Energy
Exascale Computing Project. Including a commitment to create an open source
Expand All @@ -118,18 +120,19 @@ Next, we saw the first steps towards the first iteration of Flang.

Other companies such as Arm and AMD got involved at this point.

During this time, plans to propose Classic Flang for inclusion in the LLVM Project
started to take shape.
During this time, plans were being made to propose Classic Flang for inclusion
in the LLVM Project.

* **April 2018** Steve Scalpone (NVIDIA) [announces](https://www.youtube.com/watch?v=sFVRQDgKihY)
at Euro LLVM 2018 that the frontend (which was derived from PGI's frontend)
will be re-written to address feedback from the LLVM community. This new
at Euro LLVM 2018 that the frontend of Classic Flang (which was derived from PGI's frontend)
will be rewritten to address feedback from the LLVM community. This new
front end became known as "F18".

> “We started writing this about 30 years ago, and our data structure is called integer.”
- Steve Scalpone, on the subject of the PGI frontend.

There was also a hint at a rewrite of the backend, more on this later.
There was also a hint at a rewrite of the backend, which will appear again later
as the `fir-dev` branch.
* **February 2019** Steve Scalpone (NVIDIA) [proposes](https://lists.llvm.org/pipermail/llvm-dev/2019-February/130497.html)
contributing F18 to the LLVM Project.
* **April 2019** F18 is [approved for migration](https://discourse.llvm.org/t/f18-is-accepted-as-part-of-llvm-project/51719)
Expand All @@ -151,35 +154,33 @@ started to take shape.
* **April 2020** Upstreaming of F18 into the LLVM monorepo is
[completed](https://github.com/llvm/llvm-project/commit/b98ad941a40c96c841bceb171725c925500fce6c).

At this point what was in the LLVM monorepo was F18 the rewritten frontend of
At this point what was in the LLVM monorepo was F18, the rewritten frontend of
Classic Flang. Classic Flang remained unchanged, using the PGI based frontend.

Around this time work started in the Classic Flang repo on the `fir-dev` branch
that would enable code generation when using F18.

So for the next events remember that this whole time Classic Flang was (and still
is today) in use. Its binary name was `flang`, just like the folder F18 now occupied
in the LLVM monorepo, but crucially it is a separate compiler.
is today) in use. Classic Flang's binary name was `flang`, just like the folder F18 now occupied
in the LLVM monorepo.

* **October 2020** Flang is included in an LLVM release for the first time in
LLVM 11.0.0. At this time the compiler driver was called `f18` and the aforementioned
script `flang.sh` was included as the "compiler".
LLVM 11.0.0. There was an `f18` binary and the aforementioned script
`flang.sh`.
* **October 2021** First appearance of the `flang-new` binary, in the LLVM 13.0.0
release.
* **April 2022** NVIDIA [ceases development](https://discourse.llvm.org/t/nvidia-transition-from-fir-dev/61947)
of the `fir-dev` branch, located back in the Classic Flang project. This branch
adds code generation to F18. Upstreaming of `fir-dev` to the LLVM Project
begins around this date.
of the `fir-dev` branch in the Classic Flang project. Upstreaming of `fir-dev`
to the LLVM Project begins around this date.
* **April 2022** `flang-new` can now do code generation if an experimental flag
is used. This is thanks to code being upstreamed from `fir-dev`.
is used. Enabled by the code upstreamed into the LLVM Project from `fir-dev`.
* **May 2022** The `flang.sh` driver script becomes `flang-to-external-fc`. It
allows the user to use `flang-new` to process Fortran source code then send
the result to an existing Fortran compiler such as gfortran.

This may seem strange but it allowed users to drop in Flang as a replacement
for an existing compiler and test Flang's parsing code on their own codebases
without having to wait for code generation to be finished.
* **June 2022** `flang-new` changed to do code generation
(this seems strange, but it allowed users to drop in Flang as a replacement
for an existing compiler and test Flang's parsing code on their own codebases)
* **June 2022** `flang-new` was changed to do code generation
[by default](https://github.com/llvm/llvm-project/commit/0867d4157328169c570f4d1e9c01806624035873)
with no extra flags required.
* **July 2022** NVIDIA [finish](https://discourse.llvm.org/t/nvidia-transition-from-fir-dev/61947/5)
Expand All @@ -189,13 +190,18 @@ At this point we had in the LLVM Project the Flang you see today. The
combination of the F18 frontend and MLIR based code generation from `fir-dev`.
As opposed to Classic Flang with its PGI based frontend and legacy backend.

An initiative that started with the intention of upstreaming Classic Flang had
effectively rewritten the entire compiler by the time it was complete.
An initiative to upstream Classic Flang had effectively rewritten the entire
compiler by the time it was complete.

<!-- For some reason Hugo doesn't render any plaintext in these bullet points
correctly unless I they have blank lines between them. -->

* **April 2024** The `flang-to-external-fc` script is removed. Marking the point
where `flang-new` was mature enough to be used as a standalone compiler.

* **October 2024** The community deems that Flang has met the criteria to not be
"new" and the name is changed. Goodbye `flang-new`, hello `flang`!

* TODO: First release with `flang` binary goes here!

# Flang and the Definition of New
Expand All @@ -205,7 +211,7 @@ The renaming of Flang was a contentious
and an early use of the
[LLVM proposal process](https://github.com/llvm/llvm-www/blob/main/proposals/LP0001-LLVMDecisionMaking.md).

The most critial point is that `flang` has meant different things to different
The most critial point was that `flang` has meant different things to different
people over time.

To some `flang` was the out of tree Classic Flang. To some it was the in-tree
Expand All @@ -232,20 +238,20 @@ No single list of criteria emerged but some items came up many times:
* All reasonable steps should be taken to prevent anyone using a pre-packaged
Classic Flang from confusing it with LLVM Flang.

You will see a lot of relative language there like "reasonable". It was hard
to say what that meant exactly but everyone agreed it would eventually be the case.
You will see a lot of relative language in those criteria, like "reasonable". It was hard
to say exactly what that meant everyone agreed it would eventually be the case.

Paul T Robinson summed this up well [early](https://discourse.llvm.org/t/proposal-rename-flang-new-to-flang/69462/15)
Paul T Robinson summarised the dilema [early](https://discourse.llvm.org/t/proposal-rename-flang-new-to-flang/69462/15)
in the thread:
> > the plan is to replace Classic Flang with the new Flang in the future.
>
> I suppose one of the relevant questions here is: Has the future arrived?
After which Steve Scalpone (NVIDIA) gave a
[summary](https://discourse.llvm.org/t/proposal-rename-flang-new-to-flang/69462/16)
of their perspective that the timing was not right.
After which Steve Scalpone (NVIDIA) gave
[their perspective](https://discourse.llvm.org/t/proposal-rename-flang-new-to-flang/69462/16)
their perspective that now was not the time for a name change.

We know that Flang was renamed, so what changed?
We know that Flang was renamed, so what changed since then?

* Many performance and correctness issues were addressed by the work on HLFIR.
* OpenMP support improved greatly.
Expand All @@ -261,16 +267,12 @@ We know that Flang was renamed, so what changed?
performance compared favourably with Classic Flang and was not that much behind
gfortran.

TODO: Find out if that IBM test suite link is legit.

Therefore, in October of 2024, after final discussions on the community call,
`flang-new` became `flang`. The future had arrived.
Therefore, in October of 2024 `flang-new` became `flang`. The future had arrived.

# MLIR

One advantage of the reworking of Flang between Classic Flang and LLVM Flang is
that each change allowed Flang to adopt new features from LLVM. The most notable
being MLIR.
that it allowed Flang to adopt new features from LLVM. The most notable being MLIR.

"Multi-Level Intermediate Representation", otherwise known as "MLIR" was first
[introduced](https://www.youtube.com/watch?v=qzljG6DKgic) to LLVM
Expand Down Expand Up @@ -315,7 +317,7 @@ the benefit of basing FIR on MLIR.

They did not stop there, later adding
["High Level Fortran Intermediate Representation"](https://flang.llvm.org/docs/HighLevelFIR.html)
(HLFIR) which works at a higher level than FIR. A big target of this change
(HLFIR) which works at a higher level than FIR. A big target of HLFIR
was array optimisations, which were more complex to handle using just FIR.

## IRs All the Way Down
Expand All @@ -333,11 +335,11 @@ end subroutine
[Compiler Explorer](https://godbolt.org/z/851qrcj6a))

It is tempting to think of the IRs in a "stack" where each one is converted
into the next. However, MLIR allows multiple dialects of MLIR to exist in the
into the next. However, MLIR allows multiple "dialects" of MLIR to exist in the
same file.

As you can see in this first step, Flang produces a file that uses the HLFIR
dialect where it needs to then the lower-level FIR for the rest.
dialect where it can, and the FIR dialect for the rest.

```mlir
module attributes {<...>} {
Expand All @@ -351,8 +353,9 @@ module attributes {<...>} {
}
```
For example, the parameters are declared with `hlfir.declare` but their type is
specified with `fir.array`. There is no need for HLFIR to have a `hlfir.array`
which duplicates `fir.array`.
specified with `fir.array`. MLIR's flexibility means that there is no need for
HLFIR to have a `hlfir.array` which duplicates `fir.array`, unless HLFIR wanted
to add extra information.

The next step is to convert the HLFIR parts into a combination of FIR and MLIR's
built in `arith` (arithmetic) dialect.
Expand All @@ -371,7 +374,7 @@ module attributes {<...>} {
<...>
}
```
Next this bundle of MLIR dialects is converted into LLVM IR:
Then this bundle of MLIR dialects is converted into purely LLVM IR:
```mlir
define void @foo_(ptr %0, ptr %1) {
<...>
Expand All @@ -392,12 +395,12 @@ HLFIR we have information about how the Fortran language treats those.

If we were to do this later in LLVM IR it would be much more difficult. Either
the information would be lost or incomplete, or we would be at a stage in the pipeline
where we cannot assume a specific source language.
where we cannot assume that we started with a specific source language.

# Reflections on Flang

We are almost 10 years from the first announcement of what would become LLVM
Flang. Looking at the LLVM monorepo alone there have been almost 10,000 commits
Flang. In LLVM monorepo alone there have been almost 10,000 commits
from close to 400 different contributors. Undoubtedly more in Classic Flang before
that.

Expand All @@ -423,4 +426,11 @@ might need [here](https://flang.llvm.org/docs/GettingInvolved.html).

TODO: everyone goes here

Thanks to Luis Machado (ARM) for reviewing this article.
Thanks to the following people for reviewing this article:
* Luis Machado (Arm)

Thanks to the following people for providing information for this article:
* Carlos Seo (Linaro)
* Kiran Chandramohan (Arm)
* Leandro Lupori (Linaro)
* Peter Waller (Arm)

0 comments on commit 54df17f

Please sign in to comment.