Better Scientific Software

a tutorial presented at

NOAA Global Systems Laboratory

on 9:00 am - 5:00 pm MDT (UTC-6) Tuesday 1 August 2023 - Wednesday 2 August 2023

Presenters: David E. Bernholdt (Oak Ridge National Laboratory), Anshu Dubey (Argonne National Laboratory), and Patricia A. Grubel (Los Alamos National Laboratory)

---

This page provides detailed information specific to the tutorial event above. Expect updates to this page up to, and perhaps shortly after, the date of the tutorial. Pages for other tutorial events can be accessed from the main page of this site.

---

Quick Links

• Presentation Slides (FigShare)

On this Page

• Description
• Agenda
• Presentation Slides
• How to Participate
• Stay in Touch
• Resources from Presentations
• Requested Citation
• Acknowledgements

---

Description

The BSSw tutorial focuses on issues of developer productivity, software sustainability, and reproducibility in scientific research software, particularly targeting high-performance computers.

---

Agenda

Day	Time (MDT)	Title	Presenter
Tue 1 Aug	9:00 AM	Introduction	Patricia A. Grubel (LANL)
Tue 1 Aug	9:10 AM	Motivation and Overview of Best Practices in HPC Software Development	Patricia A. Grubel (LANL)
Tue 1 Aug	9:40 AM	Collaborative Software Development	Patricia A. Grubel (LANL)
Tue 1 Aug	10:40 AM	Break
Tue 1 Aug	10:55 AM	Collaborative Software Development	Patricia A. Grubel (LANL)
Tue 1 Aug	11:25 AM	Software Packaging	David E. Bernholdt (ORNL)
Tue 1 Aug	11:55 AM	Lunch
Tue 1 Aug	1:30 PM	Scientific Software Design	Anshu Dubey (ANL)
Tue 1 Aug	3:00 PM	Refactoring Scientific Software	Anshu Dubey (ANL)
Tue 1 Aug	3:30 PM	Break
Tue 1 Aug	3:45 PM	Improving Reproducibility Through Better Software Practices	Patricia A. Grubel (LANL)
Tue 1 Aug	5:00 PM	Adjourn
Wed 2 Aug	9:00 AM	Lab Notebooks for Computational Mathematics, Sciences, & Engineering	David E. Bernholdt (ORNL)
Wed 2 Aug	10:00 AM	Managing Computational Experiments	Anshu Dubey (ANL)
Wed 2 Aug	10:30 AM	Break
Wed 2 Aug	10:45 AM	Software Testing and Verification	Anshu Dubey (ANL)
Wed 2 Aug	12:15 PM	Lunch
Wed 2 Aug	1:45 PM	Documenting your Software	David E. Bernholdt (ORNL)
Wed 2 Aug	2:45 PM	Break
Wed 2 Aug	3:00 PM	Software Licensing	David E. Bernholdt (ORNL)
Wed 2 Aug	4:00 PM	Summary	David E. Bernholdt (ORNL)
Wed 2 Aug	4:15 PM	General Discussion (optional)	All
Wed 2 Aug	5:00 PM	Adjourn

---

Presentation Slides

The latest version of the slides will always be available at https://doi.org/10.6084/m9.figshare.23796606.

Note that these files may include additional slides that will not be discussed during the tutorial, but questions are welcome.

---

How to Participate

• We want to interact with you! We find these tutorials most interesting and informative (for everyone) if you ask questions and share experiences! We learn too!

• Please raise your hand at any time to ask a question

• We have reserved extra time for discussion following many of our presentations to provide opportunities for deeper discussions of your situations and experiences

  • We encourage you to ask questions directly related to our presentations during them
  • And save the deeper discussions for after the presentation

---

Stay in Touch

• After the tutorial please feel free to email questions or feedback to the BSSw tutorial team at bssw-tutorial@lists.mcs.anl.gov.

• To find out about future events organized by the IDEAS Productivity Project, you can subscribe to our mailing list (usually ~2 messages/month).

• For monthly updates on the Better Scientific Software site, subscribe to our monthly digest.

---

Resources from Presentations

Links from the tutorial presentations are listed here for convenience

• Module 1: Introduction
  • IDEAS Productivity Project
  • The IDEAS report
  • Better Scientific Software site
  • IDEAS Productivity mailing list
  • BSSw Digest
  • BSSw RSS Feed
  • Inclusive Naming Initiative
  • Bssw.io resource on Inclusive Naming
  • Write to the tutorial authors
• Module 2: Motivation and Overview of Best Practices in HPC Software Development
  • COVID-19 epidemiology saga
    • https://doi.org/10.25561/77482
    • https://www.nicholaslewis.org/imperial-college-uk-covid-19-numbers-dont-seem-to-add-up/
    • https://www.nature.com/articles/d41586-020-01003-6
    • https://www.foxnews.com/world/imperial-college-britain-coronavirus-lockdown-buggy-mess-unreliable
    • https://www.telegraph.co.uk/technology/2020/05/16/coding-led-lockdown-totally-unreliable-buggy-mess-say-experts/
    • https://github.com/mrc-ide/covid-sim/
    • https://philbull.wordpress.com/2020/05/10/why-you-can-ignore-reviews-of-scientific-code-by-commercial-software-developers/amp/
    • http://doi.org/10.5281/zenodo.3865491
  • Best Practices for Scientific Computing
  • Good Enough Practices in Scientific Computing
  • Linux Foundation Core Infrastructure Initiative (CII) Best Practices Badging Program
  • Rate Your Project Assesment Tool
  • Progress Tracking Card (PTC) Examples
  • Productivity and Sustainability Improvement Planning
  • Better Scientific Software (BSSw)
• Module 3: Collaborative Software Development
  • Design Patterns for Git Workflows
  • Git Flow (Driessen’s Original Blog)
    • Git extensions
  • GitHub Flow (previously linked to scottchacon.com/2011/08/31/github-flow.html)
  • GitLab Flow (previously linked to docs.gitlab.com/ee/topics/gitlab_flow.html)
  • Trilinos
  • Open MPI
  • FleCSI
  • Agile Manifesto
  • How to code review in a Pull Request
  • Investing in Code Reviews for Better Research Software (previously linked to ideas-productivity.org/events/hpc-best-practices-webinars/#webinar068)
  • Testing and Code Review Practices in Research Software Development (previously linked to ideas-productivity.org/events/hpc-best-practices-webinars/#webinar044)
  • Open Source Initiative
  • Choose an Open Source License Tool
  • Introduction to Software Licensing (previously linked to ideas-productivity.org/events/hpc-best-practices-webinars/#webinar024)
• Module 4: Software Packaging
  • Simple Heat Equation Example
  • https://pyscaffold.org/
  • https://github.com/frobnitzem/lib0
  • https://code.ornl.gov/99R/mpi-test
  • https://cmake.org/cmake/help/git-stage/manual/cmake-packages.7.html#creating-packages
  • Fortran resources
    • Well documented example
    • Modern conventions example
    • Fortran standard lib (using fypp meta-programming)
    • Fortran Package Index
      • https://fortran-lang.org/
      • https://www.archaeologic.codes/software
    • Fortran Package Manager
  • https://spack.readthedocs.io
  • Anatomy of a Spack dependency “spec”
    • https://github.com/mpbelhorn/olcf-spack-environments/blob/develop/hosts/frontier/envs/base/spack.yaml
    • https://spack.readthedocs.io/en/latest/packaging_guide.html#dependency-specs
  • https://supercontainers.github.io/sc20-tutorial/
  • https://fluid-run.readthedocs.io/en/latest/HowTo/setup_your_repo.html
  • Article on CI team practices
  • https://spack.readthedocs.io/en/latest/packaging_guide.html#dependency-specs
  • DCA++
    • https://github.com/CompFUSE/DCA
    • https://github.com/twhite-cray/quip
    • https://code.ornl.gov/99R/mpi-test/-/tree/gpu_support
  • https://github.com/pyscf/extension-template
  • ZFP
    • https://github.com/LLNL/zfp
    • https://scikit-build.readthedocs.io
  • https://github.com/ECP-copa/Cabana
  • Intermediate example: C++ with spack
  • https://spack-tutorial.readthedocs.io/en/latest/tutorial_packaging.html
  • https://spack.readthedocs.io/en/latest/spack.util.html#module-spack.util.prefix
  • https://spack.readthedocs.io/en/latest/packaging_guide.html#accessing-dependencies
• Module 5: Scientific Software Design
  • Resources for independent exploration
    • Code repository in python: https://github.com/abiswas-odu/heateq-design-intersect-2023
  • Complete example in C++: https://xsdk-project.github.io/MathPackagesTraining2020/lessons/hand_coded_heat/
• Module 6: Refactoring Scientific Software
  • Refactoring example code repository: https://github.com/bssw-tutorial/hello-numerical-world
• Module 7: Improving Reproducibility Through Better Software Practices
  • Toward a Compatible Reproducibility Taxonomy for Computational and Computing Sciences
  • Reproducibility and Replicability in Science
  • Many Psychology Findings Not As Strong As Claimed
  • The War Over Supercooled Water
  • Researchers find bug in Python Script may have affected hundreds of studies
  • National Science Foundation Data Management Plan Requirements
  • Findable, Accessible, Interoperable, Re-usable
  • FAIR Data Principles US
  • SC23 Reproducibility Initiative
  • ACM Transactions on Mathematical Software (TOMS)
  • ACM Artifact Review and Badging
  • http://fursin.net/reproducibility.html
  • National Information Standards Organization (NISO) on Reproducibility and Badging
  • Helpful Tools
    • Floating Point Analysis Tools
    • Code Ocean (Cloud platforms - publish and reproduce research code and data)
    • DOIs and hosting of data, code, documents:
      • Zenodo
      • FigShare
  • Other Resources:
    • The FAIR Guiding Principles for Scientific Data Management and Stewardship. Mark D. Wilkinson, et al. 2016
    • FAIR4RS (previously linked to www.rd-alliance.org/groups/fair-research-software-fair4rs-wg)
    • Editorial: ACM TOMS Replicated Computational Results Initiative. Michael A. Heroux. 2015
    • Enhancing Reproducibility for Computational Methods
    • Simple experiments in reproducibility and technical trust by Mike Heroux and students (work in progress)
    • What every scientist should know about floating-point arithmetic. David Goldberg.
• Module 8: Lab Notebooks for Computational Mathematics, Sciences, & Engineering
  • Writing the Laboratory Notebook
  • HPC and the Lab Manager
  • What All Codes Should Do (ATPESC 2019)
  • DIKW pyramid
  • How to pick an electronic notebook
  • Resources – Execution Environments
    • ATPESC 2022 Laboratory Environment BSSw Tutorial
      • Example Repo
    • Lab Notebooks for Computational Mathematics, Sciences & Engineering (previously linked to ideas-productivity.org/events/hpc-best-practices-webinars/#webinar070)
    • Popper
    • FlashKit
    • Code Ocean
    • Weight & Biases
• Module 9: Managing Computational Experiments
  • Dubey A, Calder AC, Daley C, et al. Pragmatic optimizations for better scientific utilization of large supercomputers. The International Journal of High Performance Computing Applications. 2013;27(3):360-373. doi:10.1177/1094342012464404
  • Wilfred F. van Gunsteren, and Alan E. Mark. Validation of molecular dynamics simulation. J. Chem. Phys. 108(15), 6109-6116 (1998). doi:10.1063/1.476021
• Module 10: Software Testing and Verification
  • CI introduction
  • Useful resources on testing (formerly linked to ideas-productivity.org/resources/howtos/)
  • Lcov (formerly linked to ltp.sourceforge.net/coverage/lcov.php)
  • Additional resources:
    • Verification and Validation in Scientific Computing
    • Working Effectively with Legacy Code
    • Ongoing verification of a multiphysics community code: FLASH
• Module 11: Documenting your Software
  • API documentation tools
    • Doxygen
    • NDoc
    • Visual Expert
    • Javadoc
    • EiffelStudio
    • Sandcastle
    • ROBODoc
    • POD
    • TwinText
  • Flash-X documentation examples
    • User’s guide
    • API documentation
    • Online reference
  • Diataxis Framework
  • Markup languages
    • Markdown
    • ReStructured Text
  • Formatting/generation tools
    • Sphinx
    • Jekyll
  • Publishing/hosting sites
    • ReadtheDocs
    • GitHub Pages
• Module 12: Software Licensing
  • See Additional Resources in lesson
• Module 13: Summary
  • Productivity and Sustainability Improvement Planning
    • https://rateyourproject.org/
    • Progress Tracking Card Catalog
    • Better Scientific Software web site
  • Write to the tutorial authors
  • Tutorial Material Online
  • IDEAS Productivity project
    • mailing list
  • Better Scientific Software site
    • BSSw Digest
    • BSSw RSS Feed

---

Requested Citation

The requested citation the overall tutorial is:

David E. Bernholdt, Anshu Dubey, and Patricia A. Grubel, Better Scientific Software tutorial, in NOAA Global Systems Laboratory, Boulder, Colorado, 2023. DOI: 10.6084/m9.figshare.23796606.

Individual modules may be cited as Speaker, Module Title, in Better Scientific Software tutorial…

---

Acknowledgements

This tutorial is produced by the IDEAS Productivity project.

This work was supported by the U.S. Department of Energy Office of Science, Office of Advanced Scientific Computing Research (ASCR), and by the Exascale Computing Project (17-SC-20-SC), a collaborative effort of the U.S. Department of Energy Office of Science and the National Nuclear Security Administration.