Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

n1850.ne30_tn14.hybrid_fatessp.20241125 #61

Open
mvertens opened this issue Nov 21, 2024 · 51 comments
Open

n1850.ne30_tn14.hybrid_fatessp.20241125 #61

mvertens opened this issue Nov 21, 2024 · 51 comments
Assignees
@adagj
Labels
CLM-FATES N1850 piControl - fully coupled

Comments

@mvertens
Copy link
Contributor

mvertens commented Nov 21, 2024 •
edited by adagj
Loading

Purpose:
Duplicate of n1850.ne30_tn14.hybrid_fatessp.20241111 but with major update to the CAM code and updated CTSM/FATES code (that handles MEGAN correctly).
TODO: Not sure about the updated twostream scheme for FATES. ???
The goal is to run 20 years.

Description:
To get the code

git clone https://github.com/NorESMhub/NorESM.git
cd NorESM
git checkout noresm2_5_alpha08
./bin/git-fleximod update

 ./create_newcase --case <user_case_path>/n1850.ne30_tn14.hybrid_fatessp.20241122 \\
 --compset 1850_CAM70%LT%NORESM%CAMoslo_CLM60%FATES-SP_CICE_BLOM%ECO_MOSART_DGLC%NOEVOLVE_SWAV_SESP  \\
 --res ne30pg3_tn14 \\
 --project nn9039k --run-unsupported --mach betzy
  • user_nl_cam
 use_aerocom = .true.
 interpolate_nlat   = 192
 interpolate_nlon   = 288
 interpolate_output = .true.
 history_aerosol = .true. 
 zmconv_c0_lnd  =  0.0075D0
 zmconv_c0_ocn  =  0.0300D0
 zmconv_ke =  5.0E-6
 zmconv_ke_lnd =  1.0E-5
 clim_modal_aero_top_press =  1.D-4
 bndtvg = '/cluster/shared/noresm/inputdata/atm/cam/ggas/noaamisc.r8.nc'

  • user_nl_blom
    ICFILE = "/cluster/shared/noresm/inputdata/restart/NOIIAJRAOC20TR_TL319_tn14_ppm_20240816/rest/1775-01-01-00000/NOIIAJRAOC20TR_TL319_tn14_ppm_20240816.blom.r.1775-01-01-00000.nc"

  • user_nl_cice
    ice_ic = "/cluster/shared/noresm/inputdata/restart/NOIIAJRAOC20TR_TL319_tn14_ppm_20240816/rest/1775-01-01-00000/NOIIAJRAOC20TR_TL319_tn14_ppm_20240816.cice.r.1775-01-01-00000.nc"

  • user_nl_clm
    Add the following to user_nl_clm

fates_paramfile='/cluster/work/users/rosief/git/NorESM_MEGAN_test/components/clm/src/fates/parameter_files/fates_params_megan_twostream.nc'
z0param_method='ZengWang2007'
  • xmlchange commands
./xmlchange NTASKS=1920
./xmlchange NTASKS_OCN=256
./xmlchange ROOTPE=0
./xmlchange ROOTPE_OCN=1920
./xmlchange BLOM_VCOORD=cntiso_hybrid,BLOM_TURBULENT_CLOSURE=
./xmlchange STOP_OPTION=nyears
./xmlchange STOP_N=7
./xmlchange REST_N=1
./xmlchange REST_OPTION=nyears
./xmlchange RESUBMIT=3
./xmlchange --subgroup case.run JOB_WALLCLOCK_TIME=48:00:00
./xmlchange --subgroup case.st_archive JOB_WALLCLOCK_TIME=03:00:00

Compset long name
1850_CAM70%LT%NORESM%CAMoslo_CLM60%FATES-SP_CICE_BLOM%ECO_MOSART_DGLC%NOEVOLVE_SWAV_SESP

Model grid long name
a%ne30np4.pg3_l%ne30pg4.pg3_oi%tnx1v4_r%r05_w%null_z%null_g%gris4_m%tnx1v4

Case directory:

  • Locally on betzy:
    `/cluster/projects/nn9560k/oyvinds/NorESM2.5_alpha08/cases/n1850.ne30_tn14.hybrid_fatessp.20241125
  • On github:

Code version on github:

Diagnostics:
Atmosphere:

Output:

  • Raw output: /nird/datalake/NS9560K/noresm3/cases/n1850.ne30_tn14.hybrid_fatessp.20241125

Contacts:
@mvertens, @adagj , @oyvindseland

Extra details:
Crashed after 1 year, 11 months due to instability in cam microhysics
@mvertens mvertens added the N1850 piControl - fully coupled label Nov 21, 2024
@mvertens
Copy link
Contributor Author

@adagj - this is not ready to go yet - but I wanted to start the documentation.

@mvertens mvertens changed the title Enter casename (shortcompsetname.grid.codeversion.somekindofinfo.yyyymmdd) n1850.ne30_tn14.hybrid_fatessp.20241122 Nov 21, 2024
@rosiealice
Copy link

rosiealice commented Nov 22, 2024 •
edited
Loading

Hey @mvertens it doesn't look like I have the access to edit this issue (except for the title, weridly) but anyway, the new FATES param file with MEGAN and Twostream is in:

/cluster/work/users/rosief/git/NorESM_MEGAN_test/components/clm/src/fates/parameter_files/fates_params_megan_twostream.nc

FWIW, twostream radiation is turned on by setting this in the parameter file:
fates_rad_model = 2 ;

To recap also, we should use the version WITHOUT twostream for testing as Ryan is workig through some residual issues with mininor errors in the first timestep which stop the ERS being BFB. It is fine to run with it though. These are being fixed.
NGEET/fates#1286

@gold2718
Copy link
Contributor

@rosiealice, You should have plenty of access, does GitHub think you are logged in when you try? I often see a weird state where it knows who I am but still has the "Sign in" button at the top right of the screen.

@rosiealice
Copy link

Maybe the thing is that the issues is a 'comment' and @mvertens made that comment and maybe you can't edit someone else's comments? I am logged in... (but that does also happen a lot!!!)

@gold2718
Copy link
Contributor

maybe you can't edit someone else's comments?

That's probably it. I would guess you need to be an (evil) admin to do that sort of thing. I use the 'Quote reply' option under the three-dot drop-down on a comment to indicate the purpose of my response. If you make a text selection before doing so, it will just quote that part. Otherwise, it quotes the entire comment.

@mvertens
Copy link
Contributor Author

@adagj - noresm2_5_alpha08 has been made and I think this simulation is ready to be started. I hope those are not famous last words.

@mvertens
Copy link
Contributor Author

@adagj - please hold off for now. I'm finding some addition problems with the ctsm external. Please wait for the updated alpha08 tag which I hope to have on Monday.

@mvertens
Copy link
Contributor Author

@adagj - never mind the above message. I think we are okay with ctsm.

@mvertens
Copy link
Contributor Author

mvertens commented Nov 25, 2024 •
edited
Loading

@oyvindseland @rosiealice @kjetilaas - the clm namelists used in this run have some key differences from those used in
n1850.ne30_tn14.hybrid_fatessp.20241111.

A few key diffs:

< is n1850.ne30_tn14.hybrid_fatessp.20241111_v001
> is this simulation:

nuopc.runconfig - the mediator it looks like we are using the new gust parameterization:

<      add_gusts = .false.
>      add_gusts = .true.

drv_flds_in - we have different dust parameterizations:

<   dust_emis_method = 'Leung_2023'
>   dust_emis_method = 'Zender_2003'
>   zender_soil_erod_source = 'atm'

lnd_in:

<  stream_fldfilename_prigentroughness = '/cluster/shared/noresm/inputdata/lnd/clm2/dustemisdata/[Prigent_2005_roughness_0.25x0.25_cdf5_c240127.nc](http://prigent_2005_roughness_0.25x0.25_cdf5_c240127.nc/)'
<  stream_meshfile_prigentroughness = '/cluster/shared/noresm/inputdata/lnd/clm2/dustemisdata/[dust_0.25x0.25_ESMFmesh_cdf5_c240222.nc](http://dust_0.25x0.25_esmfmesh_cdf5_c240222.nc/)'
<  use_prigent_roughness = .true.
>  use_prigent_roughness = .false.

The above is set this way because we did not set dust_emis_method = 'Leung_2023'

>  interception_fraction = 1.0
>  maximum_leaf_wetted_fraction = 0.05
>  upplim_destruct_metamorph = 175.d00

I think what we need to do is:

  • add the following to user_nl_clm
stream_fldfilename_prigentroughness = '/cluster/shared/noresm/inputdata/lnd/clm2/dustemisdata/Prigent_2005_roughness_0.25x0.25_cdf5_c240127.nc'
stream_meshfile_prigentroughness = '/cluster/shared/noresm/inputdata/lnd/clm2/dustemisdata/dust_0.25x0.25_ESMFmesh_cdf5_c240222.nc'
use_prigent_roughness = .true.
use_z0m_snowmelt = .true.
z0param_method='Meier2022'
  • add the following to user_nl_cam
    dust_emis_method = 'Leung_2023'

NOTE: a lot of this confusion is due to the summary here: ESCOMP/CTSM#2713 (which I've read several times and still find confusing).

@oyvindseland
Copy link

Re-running the experiment with new settings.

I think I disagree with some of the conclusions/ premises in the ESCOMP. The model expert should not be the model with the most complex use of the parameter, but the model (type) that was used in developing the parameterisation. For the discussion above this means fires, Megan and dust emissions. Dry deposition of gases is dependent on parameters both in the atmosphere and on the surface. Since the processes is governed by parameters in several models then I think the complexity of the user model should decide the place of calculation.

@mvertens
Copy link
Contributor Author

@oyvindseland - thanks for your input. If I'm understanding your correctly, I agree. I think we can do things differently in noresm and do not have to replicate what is done in ESCOMP.

@kjetilaas
Copy link

For the discussion above this means fires, Megan and dust emissions. Dry deposition of gases is dependent on parameters both in the atmosphere and on the surface. Since the processes is governed by parameters in several models then I think the complexity of the user model should decide the place of calculation.

Not sure I completely understood this. Do you mean that CTSM should set fire, megan and dust emissions? And CAM set dry dep? @oyvindseland. I don't have a clear opinions on this, just want to make sure I understand.

@oyvindseland
Copy link

oyvindseland commented Nov 26, 2024 •
edited
Loading

Kjetil, yes, that are my opinion on this. At the same time my preferences are not very strong so even good technical arguments may be enough for me to change my opinion.

@oyvindseland oyvindseland changed the title n1850.ne30_tn14.hybrid_fatessp.20241122 n1850.ne30_tn14.hybrid_fatessp.20241125 Nov 26, 2024
@oyvindseland
Copy link

After discussing with @mvertens we decided to run a number of atmosphere/land tests before re-running the coupled with new settings. Will kep the issue for now, but since the model crashed I will not upload the data to nird

@rosiealice
Copy link

Which simulation was it that crashed? The one described in the initial issue? Any clues as to why?

Noting that dry dep and fire emissions are tbd still... I am going to work on that asap.

@oyvindseland
Copy link

Yes, it was this simulation. Modified the name slightly since I created it yesterday. It is unclear why it crashed after two years, but already after month 1 there seems to a drying of Asia, wettening of Sahara After a couple of year cloud cover is 99 % almost everywhere and there is hardly any cloud water in Asia and North America and quite a lot more of sea-ice.
The crash was in CLUBB

890: Error calling advance_xp2_xpyp
890: Fatal error in CLUBB: at timestep 34486
890: LAT Range: 46.7599572659205 -- 48.5538909438917
890: LON: Range: -143.877729653838 -- 0.000000000000000E+000
890: ERROR: clubb_tend_cam: Fatal error in CLUBB library
890: Image PC Routine Line Source

@mvertens
Copy link
Contributor Author

@rosiealice - we are systematically trying to see what could be causing this.
As a result, @oyvindseland is now carrying out 3 shorter simulations

  1. NF1850mam4 (this has clm60%SP)
  2. NF1850 (this also has clm60%SP)
  3. 1850_CAM70%LT%NORESM%CAMoslo_CLM60%FATESSP_CICE%PRES_DOCN%DOM_MOSART_DGLC%NOEVOLVE_SWAV_SESP

@rosiealice
Copy link

Thanks @mvertens let me know if I can help at all.

@adagj
Copy link
Contributor

adagj commented Nov 26, 2024

@oyvindseland @mvertens @rosiealice

Can you please add important results you want to share with the community in these slides: https://docs.google.com/presentation/d/1ZQnsRMfczkoCL3BaiUoxG4QBqZjwW1momM8RG2Kpds0/edit?usp=sharing

@oyvindseland
Copy link

Can do. The main problem is how to treat model crashes due to instabilities.
Just send the run catalogue to nird and run diagnostics on whatever data? Can the diagnostics work on a run catalogue or do I need to build a xxx/hist manually.

@adagj
Copy link
Contributor

adagj commented Nov 27, 2024 via email

@oyvindseland
Copy link

It is very slow / hanging when run interactively. Anyone remember how to submit it as a batch job without editing case.submit

@adagj
Copy link
Contributor

adagj commented Nov 27, 2024

It is documented here:
https://noresm-docs.readthedocs.io/en/noresm2/output/archive_output.html#short-term-archiving

@oyvindseland
Copy link

Stored at nird now.

@oyvindseland oyvindseland mentioned this issue Nov 27, 2024
Open
@oyvindseland
Copy link

oyvindseland commented Nov 27, 2024 •
edited
Loading

Yes, so I am repeating the experiment. Will create an issue, but the case as is
/cluster/projects/nn9560k/oyvinds/NorESM2.5_alpha08/cases/nf1850mam4_alpha08_ne30pg3_ne30pg3_mtn14_20241127
Also it should be mentioned that the simulation in this thread are using ZengWang2007 as defined at the start of the thread. The Leung part came in later and was suggested as a part of a restart of the experiment. Based on what was found in the atmosphere/land simulations, the oslo-aero code need to change before the alternative can be tested

@MichaelSchulzMETNO
Copy link

MichaelSchulzMETNO commented Nov 28, 2024 •
edited
Loading

comparing the 25.11. against the 11.11. run, Aeroval in 25.11. version shows almost no dust emissions, halved sea salt and DMS emissions in the first year. https://aeroval.met.no/pages/maps/?project=noresm&experiment=LatestNorESM2.5-non-observables&station=ALL&parameter=emiss&model=HYBRID-FATESSP.20241125# So considerably less wind over especially land and also ocean...

Isoprene emissions show now up over land, new feature?

@oyvindseland
Copy link

Isoprene was very low in previous versions of Fates. It looks like if it is corrected now? @kjetilaas
One detail. I guess the string [NorESM2-LM] should be removed from the timeseries legend

@kjetilaas
Copy link

kjetilaas commented Nov 28, 2024 •
edited
Loading

Isoprene was very low in previous versions of Fates. It looks like if it is corrected now? @kjetilaas

Yes! As I understand, these emissions were not sent properly in 20241111:
image

Now they are there, and geographically looks reasonable (although still an order of magnitude too small compared to NorESM2-MM-PREIND):
image

Given the large, cold bias in this simulation, I think this might be ok. But of course, we should keep an eye on this in future simulations.

@rosiealice, feel free to add more details here (or correct me, if I am wrong).

@rosiealice
Copy link

Yes, this version has MEGAN hooked up to FATES. I missed the letting as I was ok my first of four medical appointments for the morning (!) but the slide I out in the deck shows the differences in global BVOC flux totals.

@rosiealice
Copy link

rosiealice commented Nov 28, 2024 •
edited
Loading

This is for the offline model image

@oyvindseland do you have a link to the NorESM2-MM-PREIND output?

@kjetilaas
Copy link

kjetilaas commented Nov 28, 2024 •
edited
Loading

@rosiealice, see the link in the last post from Michael.

Here is the latest simulation next to NorESM2-MM-PREIND plot:
image

@rosiealice
Copy link

OK cool. What is the setup of that MM-PreInd run? I am wondering if it had much higher LAI than the FATES-SP version? Typically CLM5 has a large +ve LAI bias and that might drive these high isoprene fluxes.

@rosiealice
Copy link

... we could also test this by looking at one of the previous runs that had CLM5-SP as the land surface to check the difference caused by the switch to the FATES implementation of MEGAN? (none of those is in the Aeroval diagnostics as far as I can tell)

FWIW (@oyvindseland ) the emissions were being sent properly in the previous FATES-SP runs. The issue was that the internal leaf CO2, which is a driver of isoprene, was not passed from FATES into MEGAN (and hence it was a nonsense value and there were no isoprene emissions at all more or less).

@oyvindseland
Copy link

oyvindseland commented Nov 28, 2024 •
edited
Loading

You should be very careful drawing any conclusions from the most recent simulations since it has a very erroneous cloud cover, so very low solar radiation at the surface. 100 % cloud cover does have a large impact on LAI.

@rosiealice
Copy link

Yeah, @kjetilaas drew my attention to this. Will hold off digging too much for now.

@oyvindseland
Copy link

@rosiealice - we are systematically trying to see what could be causing this. As a result, @oyvindseland is now carrying out 3 shorter simulations

1. NF1850mam4 (this has clm60%SP)

2. NF1850 (this also has clm60%SP)

>3.1850_CAM70%LT%NORESM%CAMoslo_CLM60%FATESSP_CICE%PRES_DOCN%DOM_MOSART_DGLC%NOEVOLVE_S>WAV_SESP

The problems can also be seen in 2 and 3
3 as this simulation.
2 was completed so will upload it to Nird. (Simulation defined in its own issue)

This was referenced Nov 28, 2024
Open
Open
@adagj
Copy link
Contributor

adagj commented Nov 29, 2024 •
edited
Loading

Diagnostics for the newest AMIP type simulations:

With MAM4:

With Oslo-Aero:

And the coupled simulation:

The very high values in high clouds, ice water path, lack of tropopause ect in the fully coupled run are evident also in the AMIP-type run with Oslo-Aero, but not with MAM4

@MichaelSchulzMETNO
Copy link

Yes - it looks like it is an "Oslo-Aer in the new CAM version" problem.

@mvertens
Copy link
Contributor Author

mvertens commented Nov 30, 2024 via email

@adagj
Copy link
Contributor

adagj commented Nov 30, 2024 via email

@maritsandstad
Copy link
Collaborator

@adagj : I get 404 (File not found), but I also get that with other things on nird webpages that I have not had this issue with before, and the location and file permissions of these seem perfectly fine. I suspect this is a "something is wrong with nird web connection right now"-problem, and not a "you did something weird wrong"-problem. Maybe leave it over the Weekend and check again on Monday?

@adagj
Copy link
Contributor

adagj commented Dec 1, 2024 via email

@gold2718
Copy link
Contributor

gold2718 commented Dec 1, 2024

Hi, I don't understand why. It is in the common folder. Does anyone else have the same problem? Can Steve test?

I don't see anything either but then, https://ns2345k.web.sigma2.no/datalake shows no directories and https://ns2345k.web.sigma2.no/datalake/NS2345K gives a Not Found error. I second @maritsandstad's conclusion.

@maritsandstad
Copy link
Collaborator

@adagj , I am not Øyvind, but I'll take the mix up to be due to you being a great fan of the first Norwegian recipient of the Nobel Price for literature, and in particular his "En glad gut" (https://en.wikipedia.org/wiki/En_glad_Gut) ;-)

Thanks Øyvind. I can write to sigma. lør. 30. nov. 2024, 19:39 skrev Marit Sandstad @.>:
…
@adagj https://github.com/adagj : I get 404 (File not found), but I also get that with other things on nird webpages that I have not had this issue with before, and the location and file permissions of these seem perfectly fine. I suspect this is a "something is wrong with nird web connection right now"-problem, and not a "you did something weird wrong"-problem. Maybe leave it over the Weekend and check again on Monday? — Reply to this email directly, view it on GitHub <#61 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADZUHH4ZHGMVSCGVYDCD4UL2DIA63AVCNFSM6AAAAABSHGQQ2KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKMBZGEZTMOJXGI . You are receiving this because you were mentioned.Message ID: @.>

@MichaelSchulzMETNO
Copy link

When I looked at it two days ago the links were working, now it isnt working any longer. So its a problem for sigma2 , the webserver setup.

https://ns2345k.web.sigma2.no still works, but the link further into the directories is missing

@gold2718 gold2718 mentioned this issue Dec 1, 2024
Closed
@gold2718
Copy link
Contributor

gold2718 commented Dec 1, 2024

@oyvindseland, please see NorESMhub/CAM#183
We are working to make a new alpha tag as quickly as possible.

@oyvindseland
Copy link

@gold2718 Ongoing work noted.
Re webproblem.
I can see the files on the disk so it is likely the set-up of the webserver that is the problem.

@adagj
Copy link
Contributor

adagj commented Dec 2, 2024

@adagj , I am not Øyvind, but I'll take the mix up to be due to you being a great fan of the first Norwegian recipient of the Nobel Price for literature, and in particular his "En glad gut" (https://en.wikipedia.org/wiki/En_glad_Gut) ;-)

Thanks Øyvind. I can write to sigma. lør. 30. nov. 2024, 19:39 skrev Marit Sandstad @.>:
…
@adagj https://github.com/adagj : I get 404 (File not found), but I also get that with other things on nird webpages that I have not had this issue with before, and the location and file permissions of these seem perfectly fine. I suspect this is a "something is wrong with nird web connection right now"-problem, and not a "you did something weird wrong"-problem. Maybe leave it over the Weekend and check again on Monday? — Reply to this email directly, view it on GitHub <#61 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADZUHH4ZHGMVSCGVYDCD4UL2DIA63AVCNFSM6AAAAABSHGQQ2KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKMBZGEZTMOJXGI . You are receiving this because you were mentioned.Message ID: _@**.**_>

@maritsandstad Sorry about the mix up. I was of course reading En glad gut while replying.... will try to avoid it in the future :)

@adagj
Copy link
Contributor

adagj commented Dec 2, 2024

So, the problem with the websites seem to be fixed. Is it working for the rest of you too?

@MichaelSchulzMETNO
Copy link

all fine, works again, good !

@mvertens mvertens mentioned this issue Dec 3, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@adagj adagj

Labels
CLM-FATES N1850 piControl - fully coupled
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
8 participants
@gold2718 @mvertens @maritsandstad @rosiealice @oyvindseland @adagj @MichaelSchulzMETNO @kjetilaas