Better Software for Reproducible Science

a tutorial presented at

The International Conference for High-Performance Computing, Networking, Storage, and Analysis (SC23)

on 8:30 am - 5:00 pm MST Sunday 12 November 2023

Presenters: David E. Bernholdt (Oak Ridge National Laboratory), Patricia A. Grubel (Los Alamos National Laboratory), David M. Rogers (Oak Ridge National Laboratory), and Gregory R. Watson (Oak Ridge National Laboratory)

---

This page provides detailed information specific to the tutorial event above. Expect updates to this page up to, and perhaps shortly after, the date of the tutorial. Pages for other tutorial events can be accessed from the main page of this site.

---

Quick Links

• Presentation Slides (FigShare)

On this Page

• Description
• Agenda
• Presentation Slides
• How to Participate
• Other Software-Related Events at SC23
• Stay in Touch
• Resources from Presentations
• Requested Citation
• Acknowledgements

---

Description

Producing scientific software is a challenge. The high-performance modeling and simulation community, in particular, faces the confluence of disruptive changes in computing architectures and new opportunities (and demands) for greatly improved simulation capabilities, especially through coupling physics and scales. Simultaneously, computational science and engineering (CSE), as well as other areas of science, are experiencing an increasing focus on scientific reproducibility and software quality. Code coupling requires aggregate team interactions including integration of software processes and practices. These challenges demand large investments in scientific software development and improved practices. Focusing on improved developer productivity and software sustainability is both urgent and essential.

Attendees will learn about practices, processes, and tools to improve the productivity of those who develop CSE software, increase the sustainability of software artifacts, and enhance trustworthiness in their use. We will focus on aspects of scientific software development that are not adequately addressed by resources developed for industrial software engineering. Topics include the design, refactoring, and testing of complex scientific software systems; collaborative software development; and software packaging. The second half of this full-day tutorial will focus on reproducibility, and why and how to keep a lab notebook for computationally-based research.

---

Agenda

Time (MST)	Title	Presenter
8:30 AM	Introduction	Patricia A. Grubel (LANL)
8:40 AM	Motivation and Overview of Best Practices in HPC Software Development	Patricia A. Grubel (LANL)
9:05 AM	Scientific Software Design	David M. Rogers (ORNL)
9:30 AM	Refactoring Scientific Software	David M. Rogers (ORNL)
10:00 AM	Morning break
10:30 AM	Software Packaging	David M. Rogers (ORNL)
11:00 AM	Collaborative Software Development	Patricia A. Grubel (LANL)
12:00 PM	Lunch break
1:30 PM	Improving Reproducibility Through Better Software Practices	Gregory R. Watson (ORNL)
2:30 PM	Reproducibility of Workflows	Gregory R. Watson (ORNL)
3:00 PM	Afternoon break
3:30 PM	Software Testing and Verification	David E. Bernholdt (ORNL)
4:00 PM	Lab Notebooks for Computational Mathematics, Sciences, & Engineering	David E. Bernholdt (ORNL)
4:45 PM	Summary	David E. Bernholdt (ORNL)
5:00 PM	Adjourn

---

Presentation Slides

The latest version of the slides will always be available at https://doi.org/10.6084/m9.figshare.24226105.

Note that these files may include additional slides that will not be discussed during the tutorial, but questions are welcome.

---

How to Participate

• We want to interact with you! We find these tutorials most interesting and informative (for everyone) if you ask questions and share experiences! We learn too!

• Please raise your hand at any time to ask a question

---

Other Software-Related Events at SC23

If you’re interested in this tutorial, you might be interested in this list of other software-related events taking place in the SC23 conference.

---

Stay in Touch

• After the tutorial please feel free to email questions or feedback to the BSSw tutorial team at bssw-tutorial@lists.mcs.anl.gov.

• To find out about future events organized by the IDEAS Productivity Project, you can subscribe to our mailing list (usually ~2 messages/month).

• For monthly updates on the Better Scientific Software site, subscribe to our monthly digest.

---

Resources from Presentations

Links from the tutorial presentations are listed here for convenience

• Module 1: Introduction
  • IDEAS Productivity Project
  • The IDEAS-ECP paper preprint
  • Better Scientific Software site
  • IDEAS Productivity mailing list
  • BSSw Digest
  • RSS Feed
  • Inclusive Naming Initiative
  • Bssw.io resource on Inclusive Naming
  • Write to the tutorial authors
  • Software-related events at SC23
• Module 2: Motivation and Overview of Best Practices in HPC Software Development
  • Best Practices for Scientific Computing
  • Good Enough Practices in Scientific Computing
  • Linux Foundation Core Infrastructure Initiative (CII) Best Practices Badging Program
  • Rate Your Project Assesment Tool
  • Progress Tracking Card (PTC) Examples
  • Productivity and Sustainability Improvement Planning
  • Better Scientific Software (BSSw)
• Module 3: Scientific Software Design
  • Technical Debt in Plain English
  • Resources for independent exploration
    • Code repository in python: https://github.com/abiswas-odu/heateq-design-intersect-2023
  • Complete example in C++: https://xsdk-project.github.io/MathPackagesTraining2020/lessons/hand_coded_heat/
• Module 4: Refactoring Scientific Software
  • Refactoring example code repository: https://github.com/bssw-tutorial/hello-numerical-world
• Module 5: Software Packaging
  • Simple Heat Equation Example
  • https://pyscaffold.org/
  • https://github.com/frobnitzem/lib0
  • https://code.ornl.gov/99R/mpi-test
  • https://cmake.org/cmake/help/git-stage/manual/cmake-packages.7.html#creating-packages
  • Fortran resources
    • Well documented example
    • Modern conventions example
    • Fortran standard lib (using fypp meta-programming)
    • Fortran Package Index
      • https://fortran-lang.org/
      • https://www.archaeologic.codes/software
    • Fortran Package Manager
  • https://spack.readthedocs.io
  • Anatomy of a Spack dependency “spec”
    • https://github.com/mpbelhorn/olcf-spack-environments/blob/develop/hosts/frontier/envs/base/spack.yaml
    • https://spack.readthedocs.io/en/latest/packaging_guide.html#dependency-specs
  • https://supercontainers.github.io/sc20-tutorial/
  • https://fluid-run.readthedocs.io/en/latest/HowTo/setup_your_repo.html
  • Container Build Examples
    • https://fastapi.tiangolo.com/deployment/docker/#build-a-docker-image-for-fastapi
    • https://supercontainers.github.io/sc20-tutorial/02.docker/index.html
    • https://cloud.sylabs.io/builder
  • Article on CI team practices
  • https://spack.readthedocs.io/en/latest/packaging_guide.html#dependency-specs
  • DCA++
    • https://github.com/CompFUSE/DCA
    • https://github.com/twhite-cray/quip
    • https://code.ornl.gov/99R/mpi-test/-/tree/gpu_support
  • https://github.com/pyscf/extension-template
  • ZFP
    • https://github.com/LLNL/zfp
    • https://scikit-build.readthedocs.io
  • https://github.com/ECP-copa/Cabana
  • Intermediate example: C++ with spack
  • https://spack-tutorial.readthedocs.io/en/latest/tutorial_packaging.html
  • https://spack.readthedocs.io/en/latest/spack.util.html#module-spack.util.prefix
  • https://spack.readthedocs.io/en/latest/packaging_guide.html#accessing-dependencies
• Module 6: Collaborative Software Development
  • Design Patterns for Git Workflows
  • Git Flow (Driessen’s Original Blog)
    • Git extensions
  • GitHub Flow
  • GitLab Flow (previously linked to docs.gitlab.com/ee/topics/gitlab_flow.html)
  • Trilinos
  • Open MPI
  • FleCSI
  • Agile Manifesto
  • How to code review in a Pull Request
  • Investing in Code Reviews for Better Research Software
  • Testing and Code Review Practices in Research Software Development
  • Open Source Initiative
  • Choose an Open Source License Tool
  • Introduction to Software Licensing
• Module 7: Improving Reproducibility Through Better Software Practices
  • Toward a Compatible Reproducibility Taxonomy for Computational and Computing Sciences
  • Reproducibility and Replicability in Science
  • Many Psychology Findings Not As Strong As Claimed
  • The War Over Supercooled Water
  • Researchers find bug in Python Script may have affected hundreds of studies
  • National Science Foundation Data Management Plan Requirements
  • Findable, Accessible, Interoperable, Re-usable
  • FAIR Data Principles US
  • SC Conference Reproducibility Initiative
    • HOWTO for AEC Submitters
    • Artifact Evaluation: Tips for Authors
    • SIGOPS articles on award winning artifacts
      • https://www.sigops.org/2021/how-are-award-winning-systems-research-artifacts-prepared-part-1/
      • https://www.sigops.org/2021/how-are-award-winning-systems-research-artifacts-prepared-part-2/
    • Github CSArtifacts Resources
  • ACM Reproducible Computational Results
  • ACM Artifact Review and Badging
  • http://fursin.net/reproducibility.html
  • National Information Standards Organization (NISO) on Reproducibility and Badging
  • Helpful Tools
    • Floating Point Analysis Tools
    • Code Ocean (Cloud platforms - publish and reproduce research code and data)
    • DOIs and hosting of data, code, documents:
      • Zenodo
      • FigShare
  • Other Resources:
    • The FAIR Guiding Principles for Scientific Data Management and Stewardship. Mark D. Wilkinson, et al. 2016
    • FAIR4RS (previously linked to www.rd-alliance.org/groups/fair-research-software-fair4rs-wg)
    • Editorial: ACM TOMS Replicated Computational Results Initiative. Michael A. Heroux. 2015
    • Enhancing Reproducibility for Computational Methods
    • Simple experiments in reproducibility and technical trust by Mike Heroux and students (work in progress)
    • What every scientist should know about floating-point arithmetic. David Goldberg.
• Module 8: Reproducibility of Workflows
  • Improving Max-Min scheduling Algorithm for Reducing the Makespan of Workflow Execution in the Cloud. DOI:10.5120/ijca2017915684.
  • Example workflow systems: https://s.apache.org/existing-workflow-systems
  • Workflow repositories and registries
    • https://nf-co.re
    • https://workflowhub.eu
    • https://zenodo.org
  • Common scientific workflow patterns
    • High-performance simulation and modeling: https://doi.org/10.1007/978-3-031-23606-8_9
    • High-performance AI: https://www.analyticsvidhya.com/blog/2022/02/a-comprehensive-guide-on-hyperparameter-tuning-and-its-techniques/
    • Scientific data lifecycle: https://www.nersc.gov/assets/NERSC-10/Workflows-Archetypes-White-Paper-v1.0.pdf
    • Real-time XFEL Data Analysis at SLAC and NERSC: https://doi.org/10.48550/arXiv.2106.11469
    • Hybrid: https://doi.org/10.3389/fonc.2019.00984
  • https://opencontainers.org/
  • Containers and the Truth between HPC & Cloud System Software Convergence. 2021. DOI:10.2172/1859696
  • Other resources
    • Workflow systems
      • AiiDA: https://www.aiida.net
      • BEE: https://github.com/lanl/BEE
      • COMPSs: https://compss.bsc.es
      • Covalent: https://www.covalent.xyz
      • Cromwell: http://cromwell.readthedocs.io
      • FireWorks: https://materialsproject.github.io/fireworks
      • Galaxy: https://galaxyproject.org
      • Maesto: https://maestrowf.readthedocs.io
      • Nextflow: https://www.nextflow.io
      • Pegasus: https://pegasus.isi.edu
      • Snakemake: https://snakemake.github.io
      • Swift: http://swift-lang.org/Swift-T
      • Taskvine: https://ccl.cse.nd.edu/software/taskvine
    • Containers
      • Apptainer (singularity) https://apptainer.org
      • Charliecloud: https://hpc.github.io/charliecloud
      • Docker: https://docker.com
      • Podman: https://podman.io
      • Sarus: https://sarus.readthedocs.io/en/stable
      • Shifter: https://shifter.readthedocs.io/en/latest
    • Workflow repositories
      • @nf-core (Nextflow) https://nf-co.re
      • Snakemake workflow catalog https://snakemake.github.io/snakemake-workflow-catalog
    • Metadata frameworks
      • Common workflow language https://www.commonwl.org
      • Workflow RO-Crate https://w3id.org/workflowhub/workflow-ro-crate/1.0
      • Bioschemas Profiles https://bioschemas.org/profiles
    • Workflow repositories
      • WorkflowHub https://workflowhub.eu
      • Dockstore https://dockstore.org
    • Data repositories
      • Zenodo https://zenodo.org
      • Dataverse https://dataverse.org
    • Dynamic Provisioning and Execution of HPC Workflows Using Pythonm doi: 10.1109/PyHPC.2016.005.
    • Characterization of scientific workflows, doi:10.1109/WORKS.2008.4723958.
    • Real-Time XFEL Data Analysis at SLAC and NERSC: a Trial Run of Nascent Exascale Experimental Data Analysis, 10.48550/arXiv.2106.11469.
    • Containers and the Truth between HPC & Cloud System Software Convergence, DOI:10.2172/1859696.
• Module 9: Software Testing and Verification
  • CI introduction
  • Useful resources on testing
  • Lcov (formerly linked to ltp.sourceforge.net/coverage/lcov.php)
  • Additional resources:
    • Verification and Validation in Scientific Computing
    • Working Effectively with Legacy Code
    • Ongoing verification of a multiphysics community code: FLASH
• Module 10: Lab Notebooks for Computational Mathematics, Sciences, & Engineering
  • Writing the Laboratory Notebook
  • HPC and the Lab Manager
  • What All Codes Should Do (ATPESC 2019)
  • DIKW pyramid
  • How to pick an electronic notebook
  • Resources – Execution Environments
    • ATPESC 2022 Laboratory Environment BSSw Tutorial
      • Example Repo
    • Lab Notebooks for Computational Mathematics, Sciences & Engineering
    • Popper
    • FlashKit
    • Code Ocean
    • Weight & Biases
• Module 11: Summary
  • Productivity and Sustainability Improvement Planning
    • https://rateyourproject.org/
    • Progress Tracking Card Catalog
    • Better Scientific Software web site
  • Write to the tutorial authors
  • Tutorial Material Online
  • IDEAS Productivity project
    • mailing list
  • Better Scientific Software site
    • BSSw Digest
    • BSSw RSS Feed
  • Software-related events at SC23

---

Requested Citation

The requested citation the overall tutorial is:

David E. Bernholdt, Patricia A. Grubel, David M. Rogers, and Gregory R. Watson, Better Software for Reproducible Science tutorial, in The International Conference for High-Performance Computing, Networking, Storage, and Analysis (SC23), Denver, Colorado, 2023. DOI: 10.6084/m9.figshare.24226105.

Individual modules may be cited as Speaker, Module Title, in Better Software for Reproducible Science tutorial…

---

Acknowledgements

This tutorial is produced by the IDEAS Productivity project.

This work was supported by the U.S. Department of Energy Office of Science, Office of Advanced Scientific Computing Research (ASCR), and by the Exascale Computing Project (17-SC-20-SC), a collaborative effort of the U.S. Department of Energy Office of Science and the National Nuclear Security Administration.