Better Scientific Software

a tutorial presented at

ISC High Performance

on 2:00 pm - 6:00 pm CET Thursday 24 June 2021 - Friday 25 June 2021

Presenters: David E. Bernholdt (Oak Ridge National Laboratory), Anshu Dubey (Argonne National Laboratory), Patricia A. Grubel (Los Alamos National Laboratory), Rinku K. Gupta (Argonne National Laboratory), and David M. Rogers (Oak Ridge National Laboratory)

This page provides detailed information specific to the tutorial event above. Expect updates to this page up to, and perhaps shortly after, the date of the tutorial. Pages for other tutorial events can be accessed from the main page of this site.

Quick Links

Description

The computational science and engineering (CSE) community is in the midst of an extremely challenging period created by the confluence of disruptive changes in computing architectures, demand for greater scientific reproducibility, and new opportunities for greatly improved simulation capabilities, especially through coupling physics and scales. Computer architecture changes require new software design and implementation strategies, including significant refactoring of existing code. Reproducibility demands require more rigor across the entire software endeavor. Code coupling requires aggregate team interactions including integration of software processes and practices. These challenges demand large investments in scientific software development and improved practices. Focusing on improved developer productivity and software sustainability is both urgent and essential.

This tutorial will provide information and hands-on experience with software practices, processes, and tools explicitly tailored for CSE. Goals are improving the productivity of those who develop CSE software and increasing the sustainability of software artifacts. We discuss practices that are relevant for projects of all sizes, with emphasis on small teams, and on aggregate teams composed of small teams. Topics include software licensing, effective models, tools, and processes for small teams (including agile workflow management), reproducibility, and scientific software testing (including automated testing and continuous integration).

Agenda

Day	Time (CEST)	Module	Topic	Presenter	Time (EDT)
Thu	2:00pm-2:10pm	00	Introduction and Setup	David E. Bernholdt, ORNL	8:00am-8:10am
Thu	2:10pm-2:30pm	01	Motivation and Overview of Best Practices in HPC Software Development	David E. Bernholdt, ORNL	8:10am-8:30am
Thu	2:30pm-3:00pm	02	Agile Methodologies	Rinku Gupta, ANL	8:30am-9:00am
Thu	3:00pm-3:30pm	03	Git Workflows	Patricia A. Grubel, LANL	9:00am-9:30am
Thu	3:30pm-4:00pm	04	Scientific Software Design	Anshu Dubey, ANL	9:30am-10:00am
Thu	4:00pm-4:30pm		Break		10:00am-10:30am
Thu	4:30pm-5:00pm	05	Improving Reproducibility Through Better Software Practices	David E. Bernholdt, ORNL	10:30am-11:00am
Thu	5:00pm-5:15pm	06	Agile Methodologies Redux	Rinku Gupta, ANL	11:00am-11:15am
Thu	5:15pm-6:00pm		Hands-On Activities	All	11:15am-12:00pm

Fri	2:00pm-2:25pm	07	Software Testing Introduction	Patricia A. Grubel, LANL	8:00am-8:25am
Fri	2:25pm-2:40pm	08	Testing Walk-Through	David M. Rogers, ORNL	8:25am-8:40am
Fri	2:40pm-3:00pm	09	Testing Complex Software	David M. Rogers, ORNL	8:40am-9:00am
Fri	3:00pm-3:30pm	10	Continuous Integration	David M. Rogers, ORNL	9:00am-9:30am
Fri	3:30pm-4:00pm		Hands-On Activities	All	9:30am-10:00am
Fri	4:00pm-4:30pm		Break		10:00am-10:30am
Fri	4:30pm-5:15pm	11	Refactoring Scientific Software	Anshu Dubey, ANL	10:30am-11:15am
Fri	5:15pm-5:30pm	12	Summary	Anshu Dubey, ANL	11:15am-11:30am
Fri	5:30pm-6:00pm		Hands-On Activities	All	11:30am-12:00pm

Presentation Slides

The latest version of the slides will always be available at https://doi.org/10.6084/m9.figshare.14642520.

Note that these files may include additional slides that will not be discussed during the tutorial, but questions are welcome.

Version History:
- v2: Updates to 08-testing-walkthrough to match presented version
- v1: Initial publication

How to Participate

Please use Zoom chat to ask questions at any time. We will respond in chat or verbally as opportunities permit.
During breaks, the instructors are happy to hold further discussions with anyone interested.

Hands-On Exercises

Introduction

The hands-on exercises for this tutorial are based around a simple numerical model using the one-dimensional heat equation. The example is described briefly in the repository’s README file, and in greater detail in the ATPESC Hands-On lesson. The ATPESC version focuses on the numerical aspects of the model. But for this tutorial, we’re focused on how to make the software better from a quality perspective, so you don’t need to understand the math to do these exercises.

For the purposes of these hands-on exercises, you should imagine you’ve inherited an early version of the hello-numerical-world software from a colleague who’s left the project, and you’ve been assigned to get it into better shape so that it can be used in the next ATPESC summer school.

The repository you’ll be working with is on GitHub: bssw-tutorial/hello-numerical-world-2021-06-isc. Note: most of the screenshots will refer to the generic “hello-numerical-world” repository rather than the one specifically for this tutorial.

List of Hands-On Exercises

Note that the exercise numbers align with the presentation modules. Not every module has exercises (yet).

Thursday
- Exercise 0: Setting up the Prerequisites. Setup the accounts needed for these exercises.
- Exercise 2: Agile Methodologies. You’ll use GitHub issues and project boards to setup a simple “personal kanban” board.
- Exercise 3: Git Workflows. You’ll fork our hello-numerical-world repository, create a feature branch, and make a pull request
- Exercise 6: Agile Redux. You’ll create epic, story, and task issues for the refactoring task and track them on a kanban board
Friday
- Exercise 8: Software Testing. You’ll establish a simple continuous integration workflow and then refine it, adding code coverage assessment
- Exercise 10: Continuous Integration. You’ll establish a simple continuous integration workflow and then refine it, adding code coverage assessment
- Exercise 11: Refactoring Scientific Software. You’ll perform a small, well-defined refactoring exercise

Stay in Touch

After the tutorial please feel free to email questions or feedback to the BSSw tutorial team at bssw-tutorial@lists.mcs.anl.gov.
If you want to do the hands-on exercises, we’re happy to provide feedback on your pull requests and issues, even after the end of the tutorial.
To find out about future events organized by the IDEAS Productivity Project, you can subscribe to our mailing list (usually ~2 messages/month).
For monthly updates on the Better Scientific Software site, subscribe to our monthly digest.

Resources from Presentations

These are the links included in the tutorial presentations, included here for easier access

Module 0: Introduction and Setup
Module 1: Motivation and Overview of Best Practices in HPC Software Development
- Best Practices for Scientific Computing
- Good Enough Practices in Scientific Computing
- Linux Foundation Core Infrastructure Initiative (CII) Best Practices](https://bestpractices.coreinfrastructure.org/en) Badging Program
- Productivity and Sustainability Improvement Planning
Module 2: Agile Methodologies
Module 3: Git Workflows
- Atlassian/BitBucket (Comparing Workflows)
- Git Flow (Driessen’s Original Blog)
  - Git extensions
- GitHub Flow (previously linked to ~~scottchacon.com/2011/08/31/github-flow.html~~)
- GitLab Flow (previously linked to ~~docs.gitlab.com/ee/topics/gitlab_flow.html~~)
- Trilinos
- Open MPI
- FleCSI
Module 4: Software Design
- Related Article
- Related Webinar
Module 5: Improving Reproducibility Through Better Software Practices
- Toward a Compatible Reproducibility Taxonomy for Computational and Computing Sciences (updated 2022-03-31 with DOI link)
- Reproducibility and Replicability in Science (updated 2022-03-31 with DOI link)
- Many Psychology Findings Not as Strong as Claimed
- The War Over Supercooled Water
- Researchers Find Bug in Python Script May Have Affected Hundreds of Studies
- The FAIR Guiding Principles for Scientific Data Management and Stewardship. Mark D. Wilkinson, et al. 2016
- FAIR4RS (previously linked to ~~www.rd-alliance.org/groups/fair-research-software-fair4rs-wg~~)
- National Science Foundation Data Management Plan Requirements
- SC21 Reproducibility Initiative
- ACM Transactions on Mathematical Software (TOMS)
- ACM Artifact Review and Badging
- http://fursin.net/reproducibility.html
- National Information Standards Organization (NISO) on Reproducibility and Badging
- Floating Point Analysis Tools
- Code Ocean (Cloud platforms - publish and reproduce research code and data)
- DOIs and hosting of data, code, documents:
  - Zenodo
  - FigShare
- Editorial: ACM TOMS Replicated Computational Results Initiative. Michael A. Heroux. 2015
- Simple experiments in reproducibility and technical trust by Mike Heroux and students (work in progress)
Module 6: Agile Methodologies Redux
- none
Module 7: Testing Introduction
- Python Build and Test Framework: pyscaffold.org
- Build-Link-Test CMake Framework: llnl-blt.readthedocs.io
- Static Source Analysis (C++): clang-tidy
- Static Source Analysis (python): flake8 and pylint (updated 2022-03-31 due to dead link)
- Code Coverage Webservices: codecov and coveralls
- Tutorials for code coverage: Online Tutorial, Another example
- Development Practices Survey Article
Module 8: Testing Walk-Through
- See Hands-on activities.
Module 9: Testing Complex Applications
- Useful resources on testing (formerly linked to ~~ideas-productivity.org/resources/howtos/~~)
- Related Articles: 1, 2
Module 10: Continuous Integration
Module 11: Refactoring Scientific Software
Module 12: Summary

Requested Citation

The requested citation the overall tutorial is:

David E. Bernholdt, Anshu Dubey, Patricia A. Grubel, Rinku K. Gupta, and David M. Rogers, Better Scientific Software tutorial, in ISC High Performance, online, 2021. DOI: 10.6084/m9.figshare.14642520.

Individual modules may be cited as Speaker, Module Title, in Better Scientific Software tutorial…

Acknowledgements

This tutorial is produced by the IDEAS Productivity project.

This work was supported by the U.S. Department of Energy Office of Science, Office of Advanced Scientific Computing Research (ASCR), and by the Exascale Computing Project (17-SC-20-SC), a collaborative effort of the U.S. Department of Energy Office of Science and the National Nuclear Security Administration.