JOSE Review

[rachtorr]

Hi Jessie,
Awesome work on this! This is an interesting and well thought out workshop that I enjoyed going through, and there is a clear need shown by the marine science community. I am following the reviewer checklist and including any points that I had comments on below. I also have a few specific suggestions that are included about content in the quarto book. Feel free to reach out if you have any questions!

JOSE Checklist

Documentation

• A statement of need: Do the authors clearly state the need for this module and who the target audience is?

Need is stated in paper. For new users of the module, it would be helpful to state suggested programming level, if they need to know R and bash.

• Installation instructions: Is there a clearly stated list of dependencies?

The prerequisites page has really easy to follow steps, nice job on this.
A question I had is whether Panoply was necessary to complete all steps, or if it is optional. After working through the module it seemed optional, so maybe you can add that note in this page to be clear.

Pedagogy / Instructional design (Work-in-progress: reviewers, please comment!)

• Learning objectives: Does the module make the learning objectives plainly clear? (We don't require explicitly written learning objectives; only that they be evident from content and design.)

Expected outcomes is included in the “About” section of Quarto.
For the learning objectives section of the JOSE paper, some of the objectives could be more specific i.e. “Challenges of handling raw ESM spatial data in R” is more of a topic than learning objective. These could also include specifics like the OISST observations, Taylor Diagrams in evaluations.
The learning objectives in the paper could align with the Quarto book more - having learning goals or key topics at the beginning of each section, or being more specific to match the Chapter Index in “About”

• Content scope and length: Is the content substantial for learning a given topic? Is the length of the module appropriate?

• Pedagogy: Does the module seem easy to follow? Does it observe guidance on cognitive load? (working memory limits of 7 +/- 2 chunks of information)

The pages are split up into sections that make sense for length and for content, the flow is well structured

• Content quality: Is the writing of good quality, concise, engaging? Are the code components well crafted? Does the module seem complete?

The module seems complete! Coding components are well done

• Instructional design: Is the instructional design deliberate and apparent? For example, exploit worked-example effects; effective multi-media use; low extraneous cognitive load.

The pre-written code is helpful to follow along. Screenshots for downloading data was also super useful to follow.

JOSE paper

• A statement of need: Does the paper clearly state the need for this module and who the target audience is?

For the target audience, include a sentence to describe what level of technical experience is needed. Are learners expected to already be familiar with R and bash? Beginner level in R or intermediate?

• References: Do all archival references that should have a DOI list one (e.g., papers, datasets, software)?

A couple of references in paper are missing DOI but as other reviewer pointed out, are not peer reviewed articles

Quarto book suggestions:

Prerequisites

• Consider adding a link to the box data in the github readme, or also in Prerequisites (it was not obvious to find after I left material and came back later)
• Is panoply necessary? Add a note about this if optional
• R packages missing from installation: maps, viridis (these came up during the later modules, could be added to your list)
• Something to consider in Quarto is how to differentiate between code that goes in the terminal and code that goes in R script - even if it is directly stating it, helps the people following along on their own who may be less familiar with the different commands (this is already done in some places!) From my experience working with people who are less experienced, this can be a confusing part that requires guidance

What is an ESM?
This is a really nice overview of key concepts for ESMs, CMIP, and SSPs. I think it is smart for the purpose of the workshop to start with only two models and two scenarios, but I was wondering the reasoning behind the two models that were selected. This might be context I am missing as I was not attending the workshop, but I feel like discussion around how models and climate scenarios are selected would be important - is it because of previous studies? Do these models perform better in oceanography? How do researchers choose the number of models to include?

Download ESM output from ESGF
Following along for section 2.2 in ESGF - there were some differences in the options listed in the Quarto book and the ESFG website. I am not familiar with ESGF so it’s possible I had the wrong options selected, but I was not seeing the scenarios as described. If this is expected, you could add a note about how options could differ or may change in the future.

• For ACCESS-CM2, Experiment ID scenarios are not listed as SSP2-4.5 and SSP5-8.5 but as: 1pctCO2 (2), abrupt-4xCO2 (2), historical (2), piControl (2)
• Historical scenario not listed in IPSL-CM6A-LR option for Experiment ID
• Data from 2100+ did not show up in my system (last step in 2.3.4)

OISST

• Files in downloaded Box data have different names than the code naming in sections 4.2 and 4.3, creates an error
• Consider adding note about this message “Warning message: [rast] guessed crs”

Bias correction and downscaling
Seems like most of the steps here were done in CDO - probably not enough time in a one day workshop to get into it, but for people who are new to using this tool, include resources for learning the different commands used throughout here. I see there is a reference link in the resources section, so maybe calling attention to this early, and alerting new learners that even though you are working in R, the code is calling CDO commands through the terminal.

Evaluate accuracy of historical ESM projections
It would be helpful to adjust the names of rasters and plots in the visualization section to keep track of what is being shown when looking at the code rather than generic names like “r”, “r1”, “rr” throughout the different sections that have visualizations

[JessicaBolin]

Hi Rachel, thanks so much for your thoughtful comments and suggestions! We really appreciate the time you put into helping us improve the accessibility and clarity of the materials. We've addressed each comment below in bold italicized text. Please let us know if anything requires clarification. Thanks again for the feedback!

---

A statement of need: Do the authors clearly state the need for this module and who the target audience is?
Need is stated in paper. For new users of the module, it would be helpful to state suggested programming level, if they need to know R and bash.

• Agreed. We have added the following sentence: “This workshop is designed for marine ecologists with basic or no prior experience in climate modeling who want to incorporate Earth System Model outputs into their own ecological research using R, and no prior experience with climate modeling, Earth System Models, or Bash/shell scripting is assumed.”

Installation instructions: Is there a clearly stated list of dependencies?
The prerequisites page has really easy to follow steps, nice job on this.

A question I had is whether Panoply was necessary to complete all steps, or if it is optional. After working through the module it seemed optional, so maybe you can add that note in this page to be clear.

• Agreed. In the prerequisites page, we have amended the subtitle to “Panoply (optional)” and added a note: “Panoply is optional. All coding exercises can be completed without it, but it provides a convenient way to inspect and visualize netCDF files.”

Pedagogy / Instructional design (Work-in-progress: reviewers, please comment!)
Learning objectives: Does the module make the learning objectives plainly clear? (We don't require explicitly written learning objectives; only that they be evident from content and design.)
Expected outcomes is included in the “About” section of Quarto.

For the learning objectives section of the JOSE paper, some of the objectives could be more specific i.e. “Challenges of handling raw ESM spatial data in R” is more of a topic than learning objective. These could also include specifics like the OISST observations, Taylor Diagrams in evaluations.

• Agreed. We have revised the learning objectives in the paper to the following:

1. Introduction to ESMs (theory)
-Describe what Earth System Models (ESMs) are, how they work, and their role in marine ecological research.
-Explain the CMIP6 framework, climate scenarios, and considerations for selecting appropriate climate models.
2. Downloading ESM output (theory and live coding)
-Access and download CMIP6 model output from the Earth System Grid Federation (ESGF).
-Use shell scripts and R to efficiently download and manage large climate datasets.
3. Regridding and statistically downscaling (live coding)
-Explain why regridding and statistical downscaling are required for ESMs.
-Apply regridding and statistical downscaling methods to ESM outputs in R.
4. Observations (theory and live coding)
-Identify suitable observational datasets (e.g., OISST) for bias correction.
-Regrid and prepare observational datasets for comparison with ESM outputs.
5. Bias correction (theory and live coding)
-Explain the purpose of bias correction and the delta method.
-Apply the delta method to generate bias-corrected climate projections in R.
6. Evaluating ESM accuracy (live coding)
-Evaluate ESM performance during the observational period using quantitative metrics and visualizations (e.g., Taylor diagrams).
-Interpret model performance to inform model selection for ecological applications.
7. Making projections (live coding)
-Generate future sea-surface temperature projections from the two CMIP6 models and two climate scenarios.
-Visualize projected changes through time series plots and spatial maps.

The learning objectives in the paper could align with the Quarto book more - having learning goals or key topics at the beginning of each section, or being more specific to match the Chapter Index in “About”

• Agreed. We have added learning objective callout boxes at the beginning of each chapter in the Quarto book to clearly outline the learning objectives covered. These learning objectives are now aligned with the chapter content and the Chapter Index in the "About" section.

Content scope and length: Is the content substantial for learning a given topic? Is the length of the module appropriate?
Pedagogy: Does the module seem easy to follow? Does it observe guidance on cognitive load? (working memory limits of 7 +/- 2 chunks of information)
The pages are split up into sections that make sense for length and for content, the flow is well structured

• Thank you!

Content quality: Is the writing of good quality, concise, engaging? Are the code components well crafted? Does the module seem complete?
The module seems complete! Coding components are well done

• Thank you!

Instructional design: Is the instructional design deliberate and apparent? For example, exploit worked-example effects; effective multi-media use; low extraneous cognitive load.
The pre-written code is helpful to follow along. Screenshots for downloading data was also super useful to follow.

• Thank you!

JOSE paper

A statement of need: Does the paper clearly state the need for this module and who the target audience is?
For the target audience, include a sentence to describe what level of technical experience is needed. Are learners expected to already be familiar with R and bash? Beginner level in R or intermediate?

• Agreed. We have addressed this point above, and also for the other reviewer’s comment.

References: Do all archival references that should have a DOI list one (e.g., papers, datasets, software)?
A couple of references in paper are missing DOI but as other reviewer pointed out, are not peer reviewed articles

• We have removed both of these references from the paper.

Quarto book suggestions:

Prerequisites

Consider adding a link to the box data in the github readme, or also in Prerequisites (it was not obvious to find after I left material and came back later)

• Done. Have added links the data required in the README, and also in the prerequisites via a callout box.

Is panoply necessary? Add a note about this if optional

• Agreed. We have addressed this point above in the second comment.

R packages missing from installation: maps, viridis (these came up during the later modules, could be added to your list)

• Thanks! Fixed.

Something to consider in Quarto is how to differentiate between code that goes in the terminal and code that goes in R script - even if it is directly stating it, helps the people following along on their own who may be less familiar with the different commands (this is already done in some places!) From my experience working with people who are less experienced, this can be a confusing part that requires guidance

• This is great advice, thank you! We have updated the materials to more clearly distinguish between terminal and R code. We have added explicit headings (#### In Terminal and #### In R) before the relevant code chunks on the Prerequisites page, making it clearer which environment each command should be run in.

What is an ESM?

This is a really nice overview of key concepts for ESMs, CMIP, and SSPs. I think it is smart for the purpose of the workshop to start with only two models and two scenarios, but I was wondering the reasoning behind the two models that were selected. This might be context I am missing as I was not attending the workshop, but I feel like discussion around how models and climate scenarios are selected would be important - is it because of previous studies? Do these models perform better in oceanography? How do researchers choose the number of models to include?

• Thanks - we agree. We decided to intentionally limit the examples to two ESMs and two climate scenarios to keep the computational requirements manageable and allow the participants to focus on learning the workflow rather than comparing model performance. We’ve added text clarifying that these models were selected as representative examples for demonstration only on the “What is an ESM” page: “These models were selected as representative examples for demonstrating the workflow in this workshop and should not be interpreted as recommended or best-performing models. In your own research, ESM selection is typically guided by the study objectives, region, variable(s) of interest, computational resources, and published model evaluation studies.” We also discuss model selection in greater detail during the workshop's theory session using the accompanying PowerPoint slides (ESMRworkshop_theory.pptx), which is in the root directory of the Github repo.

Download ESM output from ESGF

Following along for section 2.2 in ESGF - there were some differences in the options listed in the Quarto book and the ESFG website. I am not familiar with ESGF so it’s possible I had the wrong options selected, but I was not seeing the scenarios as described. If this is expected, you could add a note about how options could differ or may change in the future.
For ACCESS-CM2, Experiment ID scenarios are not listed as SSP2-4.5 and SSP5-8.5 but as: 1pctCO2 (2), abrupt-4xCO2 (2), historical (2), piControl (2)

• Thanks for catching this and correct - the available options depend on the selected ActivityID. If CMIP is selected, only the historical and idealized experiments are displayed. The ActivityID needs to e changed to ScenarioMIP in order to see the SSP options. We’ve added text in Section 2.2.4 to clarify this distinction.

Historical scenario not listed in IPSL-CM6A-LR option for Experiment ID

• See above comment.

Data from 2100+ did not show up in my system (last step in 2.3.4)

• We’ve added the following text in 2.3.4 to clarify how options may differ in future: “If your scripts have downloaded data for 2100-2300 (i.e., some models will do this, but not all, and is subject to change)...”

OISST

Files in downloaded Box data have different names than the code naming in sections 4.2 and 4.3, creates an error

• Thanks! We’ve fixed this error and the files are now consistently named.

Consider adding note about this message “Warning message: [rast] guessed crs”

• Done. We’ve added the following text in 4.3: “The warning Warning message: [rast] guessed crs may appear in the console for some users - this is expected in this context, and can be safely ignored.”

Bias correction and downscaling

Seems like most of the steps here were done in CDO - probably not enough time in a one day workshop to get into it, but for people who are new to using this tool, include resources for learning the different commands used throughout here. I see there is a reference link in the resources section, so maybe calling attention to this early, and alerting new learners that even though you are working in R, the code is calling CDO commands through the terminal.

• Thanks for the suggestion. We’ve added a tip callout at the beginning of the page clarifying this, and directing users to the Resources page for more information on CDO.

Evaluate accuracy of historical ESM projections

It would be helpful to adjust the names of rasters and plots in the visualization section to keep track of what is being shown when looking at the code rather than generic names like “r”, “r1”, “rr” throughout the different sections that have visualizations

• Agreed and we have updated all generic names in this section (e.g., “r1”, “r2” etc) to descriptive names.

[rachtorr]

Thanks @JessicaBolin and great work making these quick updates!
I didn't realize the powerpoint was included, but seeing those slides helped clarify some of the conceptual questions I had about model selection.