November 17 2022

Agenda

• New workgroup co-lead: Fahim Sidi (US EPA)
• CMAS cloud computing session recap
• Round table updates on cloud computing from the group
• Public data archives on the cloud
• Call frequency: shift to every other month?

Notes

CMAS cloud computing session recap

• CMAS session on cloud computing
• Presentations
  • Performance Optimization of the Community Multiscale Air Quality (CMAQ) model on Microsoft Azure
    • Focus on new and proprietary hardware @ MS, and process pinning to optimize memory/computing on specific versions of Epyc processors (Milan X)
  • Community Multiscale Air Quality (CMAQ) Modeling and Analysis on the Cloud
    • Presented this work to the group in the past months
    • Interested in hearing from the community about applications to gain support from the providers for the community
  • Scaling NWP workloads on AWS to achieve your research goals
  • Use of NOAAs Global Forecast System Data in the Cloud for Community Air Quality Modeling
  • Open-innovation and Open-development Framework for the Unified Forecastsystem - An EPIC Approach
  • CMAS tutorials on running CMAQ on AQS and Azure
    • If users go through the steps in the online tutorials, they will be able to spin up open access processors and run a test case of the models
    • CMAS will offer monthly help sessions (1st Friday), user support
    • Looking for feedback on the tutorials

Round table updates on cloud computing from the group

• Zac Adelman (LADCO)
  • Preparing for production modeling in support of ozone attainment strategies
  • Will have more data to report in the next few months on cost/performance
• Weining Zhao (TX)
  • Testing CMAQ and CAMx on different cloud service providers
  • Data transfer and costs tests
  • Gathering data to choose which cloud provider they will use
  • Running into some issues related to forecasting charges; because they're in the state, they have to go through a state service, which charges an overhead service fee, and their fee is unpredictable; the fee is ~ 100% of the cloud provider fees; not cost effective
  • Goal is not to completely replace in-house servers, but to have additional resources to scale as needed
• Fahim Sidi (EPA)
  • Collaborating with CMAS on documenting CMAQ on the cloud with AWS and Azure; includes tutorials and documentation; informing development at EPA
  • Working with AWS on their Open Data program for storing and accessing large datasets with no egress fees
  • Seeking feedback on what would be helpful in the second phase; e.g., updating CMAQ on the tutorials, post-processing, AMET, VERDI, SMOKE, subsetting domains/analysis workflow
• Byeong Kim (GA)
  • 2 instances up and running for testing; moving towards production mode; small instance for development and scripting; large instance for bigger compute jobs
  • Similar challenges as TX with state agency overhead charges; their charges are based on per instance, complicated by using big HPC clusters with many instances
  • On Amazon servers EBS is the default on the head node; EBS storage will be wiped out if you terminate the instance unless you specify that it doesn't get wiped out
  • External storage: Amazon Snowcone for data distribution can't be connected via USB, has to be used through WIFI or Ethernet
• Discussion around CMAS developing a modeling instance for production
  • CMAS thinking through design: model versions, maintenance, several different mount points for each piece of software vs 1 mount point

Public data archives on the cloud

• Which groups out there are serving data on the cloud?
• How to find the catalogs and access these data?
• Talk about this on the next call

Call frequency: shift to every other month?

• Next call January 19, then every other month after

Next/Future Meeting Ideas

• Reach out to Patrick Campbell to give his CMAS presentation to this group in the spring