Developing a Testing and Continuous Integration Strategy for your Team

a tutorial presented at

Exascale Computing Project Annual Meeting

on 2:30 pm - 4:00 pm EDT Monday 12 April 2021

Presenters: David E. Bernholdt (Oak Ridge National Laboratory), Patricia A. Grubel (Los Alamos National Laboratory), and James M. Willenbring (Sandia National Laboratories)

This page provides detailed information specific to the tutorial event above. Expect updates to this page up to, and perhaps shortly after, the date of the tutorial. Pages for other tutorial events can be accessed from the main page of this site.

Quick Links

Playlist (YouTube)
Presentation Slides (FigShare)

Description

A thorough and robust testing regime is central to the productive development, evolution, and maintenance of quality, trustworthy scientific software. Continuous integration, though much discussed, is just one element of such a testing regime. Most project teams feel that they could (and should) do a “better job” of testing. In many cases, designing and implementing a strong testing strategy can seem so daunting that it is hard to know where to start.

In this tutorial, which is aimed at those with beginner to intermediate levels of comfort with testing and continuous integration, we will briefly review the multiple motivations for testing, and the different types of tests that address them. We’ll discuss some strategies for testing complex software systems, and how continuous integration testing fits into the larger picture. Accompanying hands-on activities, available for self-study, will demonstrate how to get started with a very simple level of CI testing.

Agenda

Time (EDT)	Module	Topic	Speaker
2:30pm-2:35pm	00	Introduction	David E. Bernholdt, ORNL
2:35pm-2:40pm	01	Motivation and Overview	Patricia A. Grubel, LANL
2:40pm-3:00pm	02	Software Testing 1	Patricia A. Grubel, LANL
3:00pm-3:25pm	03	Software Testing 2	David E. Bernholdt, ORNL
3:25pm-3:55pm	04	Continuous Integration	James M. Willenbring, SNL
3:55pm-4:00pm	05	Summary	James M. Willenbring, SNL

Presentation Slides

The latest version of the slides will always be available at https://doi.org/10.6084/m9.figshare.14376956.

Note that these files may include additional slides that will not be discussed during the tutorial, but questions are welcome.

How to Participate

Please remember to stay muted except when asking questions, to limit the extraneous noise.
Please use the Zoom chat to ask questions at any time. We’ll try to respond to clarifications promptly. As time permits, we’ll take questions at the ends of presentations through chat (including responding to earlier questions) or verbally.

You may be interested in other sessions at this year’s ECP Annual Meeting related to the IDEAS project, software process improvement, testing, and continuous integration.

IDEAS Desk – informal conversations about developer productivity and software sustainability
- Room 125 In the Gather.Town Side Meetings area
- 12:30pm-2:00pm Monday through Friday
Validation, Verification and Performance Suites, Proxy Apps, and Continuous Integration for Exascale Supercomputers
- 10:00am-11:30am Wednesday
Advancing Scientific Productivity through Better Scientific Software
- 11:30am-12:30am Wednesday
Hands-on with Progress Tracking Cards: A Lightweight Method for Improving Your Software Practices
- 1:00pm-3:30pm Wednesday
Benefiting from ECP CI
- 2:30pm-3:30pm Wednesday
ECP CI Startup Tutorial
- 2:30pm-6:00pm Friday

Stay in Touch

After the tutorial please feel free to email questions or feedback to the BSSw tutorial team at bssw-tutorial@lists.mcs.anl.gov.
If you want to do the hands-on exercises, we’re happy to provide feedback on your pull requests and issues, even after the end of the tutorial.
To find out about future events organized by the IDEAS Productivity Project, you can subscribe to our mailing list (usually ~2 messages/month).
For monthly updates on the Better Scientific Software site, subscribe to our monthly digest.

Resources from Presentations

These are the links included in the tutorial presentations, included here for easier access

Module 0: Introduction
Module 1: Motivation and Overview of Best Practices in HPC Software Development
- none
Module 2: Software Testing 1
- Python Build and Test Framework: pyscaffold.org
- Build-Link-Test CMake Framework: llnl-blt.readthedocs.io
- Static Source Analysis (C++): clang-tidy
- Static Source Analysis (python): flake8 and pylint (updated 2022-03-31 due to dead link)
- Code Coverage Webservices: codecov and coveralls
- Tutorials for code coverage: Online Tutorial, Another example
- Development Practices Survey Article
Module 3: Software Testing 2
- Useful resources on testing (formerly linked to ~~ideas-productivity.org/resources/howtos/~~)
- Related Articles: 1, 2
Module 4: Continuous Integration
Module 5: Summary
- Productivity and Sustainability Improvement Planning
- Better Scientific Software web site

Requested Citation

The requested citation the overall tutorial is:

David E. Bernholdt, Patricia A. Grubel, and James M. Willenbring, Developing a Testing and Continuous Integration Strategy for your Team tutorial, in Exascale Computing Project Annual Meeting, online, 2021. DOI: 10.6084/m9.figshare.14376956.

Individual modules may be cited as Speaker, Module Title, in Developing a Testing and Continuous Integration Strategy for your Team tutorial…

Acknowledgements

This tutorial is produced by the IDEAS Productivity project.

This work was supported by the U.S. Department of Energy Office of Science, Office of Advanced Scientific Computing Research (ASCR), and by the Exascale Computing Project (17-SC-20-SC), a collaborative effort of the U.S. Department of Energy Office of Science and the National Nuclear Security Administration.