Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OpenACC port of wave propagation (ACCV6) #6

Merged
merged 31 commits into from
Feb 22, 2024
Merged

OpenACC port of wave propagation (ACCV6) #6

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
ae322e0
ACCV6
fdisante Nov 17, 2023
bc824f1
Update intel-oneapi install script
wdeconinck Nov 20, 2023
756a754
Some comments and nvtx removed, field_api destructor restored
fdisante Dec 7, 2023
d1bd955
removed comments in propags2
fdisante Dec 10, 2023
6c46fa8
Merge pull request #1 from fdisante1/naan-phys-gpu
fdisante Dec 10, 2023
b555fec
removed comments and guarded use openacc propag_wam - removed/added s…
fdisante Dec 10, 2023
4f6d4f1
removed unused acc data from wamintgr_loki
fdisante Dec 10, 2023
de24c4c
ctuwdrv: restored comment
fdisante Dec 10, 2023
a6090b1
ctuwdrv: restored whitespaces
fdisante Dec 10, 2023
3558443
ctuwdrv: restored whitespaces
fdisante Dec 10, 2023
31a263a
ctuwdrv: restored whitespaces
fdisante Dec 10, 2023
a0d568a
ctuwini: removed commented module, restored dr_hook
fdisante Dec 10, 2023
c655ed2
ctuwupdt: removed commented openacc pragmas
fdisante Dec 10, 2023
dd87579
ctuwupdt: removed commented openacc pragmas and !F
fdisante Dec 10, 2023
d642d66
wamintgr_loki_gpu: FIELD_API destructors restored
fdisante Dec 10, 2023
ed6a9d1
wamintgr_loki_gpu: whitespace removed
fdisante Dec 10, 2023
79988d1
ctuwini: uncommented ZHOOK_HANDLE declaration
fdisante Dec 11, 2023
142659b
ctuwupdt: removed unused module, propag_wam: fixed openacc data pragma
fdisante Dec 11, 2023
6b8301a
ctuw: reverted spaces
fdisante Dec 11, 2023
a8f5c9b
ctuwini: uncommented DR_HOOK reverted spaces
fdisante Dec 11, 2023
a0c9607
cuda aware MPI
fdisante Jan 17, 2024
f78680f
CPU-only ecWAM variant fails validation solved
fdisante Jan 18, 2024
7084540
Using MTHREADS = OMP_GET_MAX_THREADS when openacc activated
fdisante Jan 29, 2024
b7dd9c9
Removed unclosed acc enter data and converted in acc declare
fdisante Jan 29, 2024
d74d1f6
Cleaning code (removing spaces and comments) updated the timestep for…
fdisante Feb 1, 2024
4262014
More cleaning and minor fixing. OMP_GET_MAX_THREADS
fdisante Feb 1, 2024
e422898
removed trailing spaces in the ifdef endif to make gnu compiling succ…
fdisante Feb 1, 2024
fb87ad5
removed use openacc (not needed) to make intel compiling succesfully
fdisante Feb 1, 2024
318594c
proenvhalo ported to GPU
fdisante Feb 8, 2024
d026989
use MTHREADS=1 when OPENACC is activated
fdisante Feb 12, 2024
54e56ef
CTUW: make preprocessor flags lower case
awnawab Feb 14, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
297 changes: 207 additions & 90 deletions src/ecwam/ctuw.F90
View file
Open in desktop

Large diffs are not rendered by default.

16 changes: 15 additions & 1 deletion src/ecwam/ctuwdrv.F90
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,12 +29,18 @@ SUBROUTINE CTUWDRV (DELPRO, MSTART, MEND, &
USE PARKIND_WAVE, ONLY : JWIM, JWRB, JWRU
USE YOWDRVTYPE , ONLY : WVGRIDGLO


USE YOWCURR , ONLY : LLCFLCUROFF
USE YOWGRID , ONLY : NPROMA_WAM
USE YOWMPP , ONLY : IRANK
USE YOWPARAM , ONLY : NIBLO ,NANG ,NFRE_RED
USE YOWSTAT , ONLY : IREFRA
USE YOWTEST , ONLY : IU06
USE YOWUBUF , ONLY : WLATN ,WLONN ,WCORN
USE YOWFRED , ONLY : FR ,DELTH, COSTH ,SINTH
USE YOWPCONS , ONLY : ZPI



USE YOMHOOK , ONLY : LHOOK, DR_HOOK, JPHOOK

Expand Down Expand Up @@ -73,12 +79,15 @@ SUBROUTINE CTUWDRV (DELPRO, MSTART, MEND, &

IF (LHOOK) CALL DR_HOOK('CTUWDRV',0,ZHOOK_HANDLE)

!! NPROMA=NPROMA_WAM
!! =NPROMA_WAM
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can this comment please be restored to its original state?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

MTHREADS=1
!$ MTHREADS=OMP_GET_MAX_THREADS()
NPROMA=(IJL-IJS+1)/MTHREADS + 1


#ifndef _OPENACC
!$OMP PARALLEL DO SCHEDULE(DYNAMIC,1) PRIVATE(JKGLO, KIJS, KIJL, ICALL, IJ, LL2NDCALL)
#endif /*_OPENACC*/
DO JKGLO = IJS, IJL, NPROMA
KIJS=JKGLO
KIJL=MIN(KIJS+NPROMA-1,IJL)
Expand All @@ -91,6 +100,7 @@ SUBROUTINE CTUWDRV (DELPRO, MSTART, MEND, &
& COSPHM1_EXT, DEPTH_EXT, U_EXT, V_EXT )



Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please revert this whitespace change.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

! WHEN SURFACE CURRENTS ARE USED AND LLCFLCUROFF IS TRUE
! THEN TRY TO SATISFY THE CFL CONDITION WITHOUT THE CURRENTS
! IF IT WAS VIOLATED IN THE FIRST PLACE
Expand All @@ -112,8 +122,12 @@ SUBROUTINE CTUWDRV (DELPRO, MSTART, MEND, &
& COSPHM1_EXT, DEPTH_EXT, U_EXT, V_EXT )
ENDIF
ENDIF

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please revert this whitespace change.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done


ENDDO
#ifndef _OPENACC
!$OMP END PARALLEL DO
#endif /*_OPENACC*/

DO IJ=IJS,IJL
IF (LCFLFAIL(IJ)) THEN
Expand Down
36 changes: 28 additions & 8 deletions src/ecwam/ctuwini.F90
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,8 @@
! granted to it by virtue of its status as an intergovernmental organisation
! nor does it submit to any jurisdiction.
!
!MODULE CTUWINI_MOD
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please remove the commented out MODULE statements?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

! CONTAINS

SUBROUTINE CTUWINI (KIJS, KIJL, NINF, NSUP, BLK2GLO, COSPHM1_EXT, &
& WLATM1, WCORM1, DP)
Expand Down Expand Up @@ -44,18 +46,20 @@ SUBROUTINE CTUWINI (KIJS, KIJL, NINF, NSUP, BLK2GLO, COSPHM1_EXT, &
REAL(KIND=JWRB), DIMENSION(NINF:NSUP,4), INTENT(OUT) :: WCORM1 ! 1 - WCOR
REAL(KIND=JWRB), DIMENSION(NINF:NSUP,2), INTENT(OUT) :: DP ! COS PHI FACTOR


INTEGER(KIND=JWIM) :: IJ, K, M, IC, ICR, ICL, KY, KK, KKM
INTEGER(KIND=JWIM) :: NLAND

REAL(KIND=JPHOOK) :: ZHOOK_HANDLE
!REAL(KIND=JPHOOK) :: ZHOOK_HANDLE

!!$acc routine vector

! ----------------------------------------------------------------------

IF (LHOOK) CALL DR_HOOK('CTUWINI',0,ZHOOK_HANDLE)
!IF (LHOOK) CALL DR_HOOK('CTUWINI',0,ZHOOK_HANDLE)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why have the DR_HOOK calls been removed?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reverted


NLAND = NSUP+1


!$acc parallel loop independent collapse(2)
DO IC=1,2
DO IJ = KIJS,KIJL
IF (KLAT(IJ,IC,1) < NLAND .AND. KLAT(IJ,IC,2) < NLAND) THEN
Expand All @@ -74,7 +78,9 @@ SUBROUTINE CTUWINI (KIJS, KIJL, NINF, NSUP, BLK2GLO, COSPHM1_EXT, &
ENDIF
ENDDO
ENDDO

!$acc end parallel

!$acc parallel loop independent collapse(2)
DO ICR=1,4
DO IJ = KIJS,KIJL
IF (KCOR(IJ,ICR,1) < NLAND .AND. KCOR(IJ,ICR,2) < NLAND) THEN
Expand All @@ -88,15 +94,17 @@ SUBROUTINE CTUWINI (KIJS, KIJL, NINF, NSUP, BLK2GLO, COSPHM1_EXT, &
ELSE
! ADAPT CORNER POINT INTERPOLATION WEIGHT IF LAND IS PRESENT
! SECOND CLOSEST CORNER POINT IS OVER LAND
IF (WCOR(IJ,ICR) > 0.5_JWRB) WCOR(IJ,ICR)=1.0_JWRB
IF (WCOR(IJ,ICR) > 0.5_JWRB) WCOR(IJ,ICR)=1.0_JWRB
WCORM1(IJ,ICR) = 1.0_JWRB - WCOR(IJ,ICR)
ENDIF
ENDDO
ENDDO
!$acc end parallel


! INITIALISATION

!$acc parallel loop independent collapse(5)
DO ICL=1,2
DO IC=1,2
DO M=1,NFRE_RED
Expand All @@ -108,7 +116,10 @@ SUBROUTINE CTUWINI (KIJS, KIJL, NINF, NSUP, BLK2GLO, COSPHM1_EXT, &
ENDDO
ENDDO
ENDDO
!$acc end parallel


!$acc parallel loop independent collapse(4)
DO IC=1,2
DO M=1,NFRE_RED
DO K=1,NANG
Expand All @@ -118,7 +129,10 @@ SUBROUTINE CTUWINI (KIJS, KIJL, NINF, NSUP, BLK2GLO, COSPHM1_EXT, &
ENDDO
ENDDO
ENDDO
!$acc end parallel


!$acc parallel loop independent collapse(5)
DO ICL=1,2
DO ICR=1,4
DO M=1,NFRE_RED
Expand All @@ -130,26 +144,32 @@ SUBROUTINE CTUWINI (KIJS, KIJL, NINF, NSUP, BLK2GLO, COSPHM1_EXT, &
ENDDO
ENDDO
ENDDO
!$acc end parallel



IF (ICASE == 1) THEN

!* SPHERICAL GRID.
! ---------------

!
!* COMPUTE COS PHI FACTOR FOR ADJOINING GRID POINT.
! (for all grid points)
!$acc parallel loop independent collapse(2) private(KY,KK,KKM)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we please guard openacc parallel clauses behind ifdefs?

DO IC=1,2
! !!!$acc loop private(KY,KK,KKM)
DO IJ = KIJS,KIJL
KY=BLK2GLO%KXLT(IJ)
KK=KY+2*IC-3
KKM=MAX(1,MIN(KK,NGY))
DP(IJ,IC) = COSPH(KKM)*COSPHM1_EXT(IJ)
ENDDO
ENDDO
!$acc end parallel
ENDIF

IF (LHOOK) CALL DR_HOOK('CTUWINI',1,ZHOOK_HANDLE)

!IF (LHOOK) CALL DR_HOOK('CTUWINI',1,ZHOOK_HANDLE)

END SUBROUTINE CTUWINI
!END MODULE CTUWINI_MOD
Loading
Loading