Developing a Testing and Continuous Integration Strategy for your Team

a tutorial presented at

Exascale Computing Project Annual Meeting

on 2:30 pm - 4:00 pm EDT (UTC-4) Monday 2 May 2022

Presenters: David M. Rogers (Oak Ridge National Laboratory) and Gregory R. Watson (Oak Ridge National Laboratory)

This page provides detailed information specific to the tutorial event above. Expect updates to this page up to, and perhaps shortly after, the date of the tutorial. Pages for other tutorial events can be accessed from the main page of this site.

Quick Links

Description

A thorough and robust testing regime is central to the productive development, evolution, and maintenance of quality, trustworthy scientific software. Continuous integration, though much discussed, is just one element of such a testing regime. Most project teams feel that they could (and should) do a “better job” of testing. In many cases, designing and implementing a strong testing strategy can seem so daunting that it is hard to know where to start.

In this tutorial, which is aimed at those with beginner to intermediate levels of comfort with testing and continuous integration, we will briefly review the multiple motivations for testing, and the different types of tests that address them. We’ll discuss some strategies for testing complex software systems, and how continuous integration testing fits into the larger picture. Accompanying hands-on activities, available for self-study, will demonstrate how to get started with a very simple level of CI testing.

Agenda

Time (EDT)	Module	Title	Presenter
2:30 PM	0	Introduction and Setup	Gregory R. Watson (ORNL)
2:35 PM	1	Motivation and Overview	Gregory R. Watson (ORNL)
2:40 PM	2	Software Testing Introduction	Gregory R. Watson (ORNL)
3:00 PM	3	Testing Complex Software	David M. Rogers (ORNL)
3:25 PM	4	Continuous Integration	David M. Rogers (ORNL)
3:55 PM	5	Summary	David M. Rogers (ORNL)
4:00 PM		Adjourn

Presentation Slides

The latest version of the slides will always be available at https://doi.org/10.6084/m9.figshare.19608927.

Note that these files may include additional slides that will not be discussed during the tutorial, but questions are welcome.

How to Participate

Please use Zoom chat or unmute to ask questions at any time. We will respond in chat or verbally as opportunities permit.

Hands-On Exercises

Introduction

The hands-on exercises for this tutorial are based around a simple numerical model using the one-dimensional heat equation. The example is described briefly in the repository’s README file, and in greater detail in the ATPESC Hands-On lesson. The ATPESC version focuses on the numerical aspects of the model.

But for this tutorial, we’re focused on how to make the software better from a quality perspective, so you don’t need to understand the math to do these exercises.

For the purposes of these hands-on exercises, you should imagine you’ve inherited an early version of the hello-numerical-world software from a colleague who’s left the project, and you’ve been assigned to get it into better shape so that it can be used in the next ATPESC summer school.

The repository you’ll be working with is: bssw-tutorial/hello-numerical-world-2022-05-02-ecpam.

Note: most of the screenshots will refer to the generic “hello-numerical-world” repository rather than the one specifically for this event.

List of Hands-On Exercises

Note that not every presentation module has exercises (yet).

Setting up the Prerequisites. Setup the accounts needed for these exercises.
Software Testing. You’ll use an example project to try out using test driven development to add new functionality to a project
Continuous Integration. You’ll establish a simple continuous integration workflow and then refine it, adding code coverage assessment

There are also activities associated with a couple of modules that we didn’t have time to cover in this tutorial. You are welcome to try them out too.

Agile Methodologies. You’ll use GitHub issues and project boards to setup a simple “personal kanban” board.
Basic Git for Collaboration. You’ll fork our hello-numerical-world repository, create a feature branch, and make a pull request
Agile Redux. You’ll create epic, story, and task issues for the refactoring task and track them on a kanban board
Refactoring Scientific Software. You’ll perform a small, well-defined refactoring exercise

Stay in Touch

After the tutorial please feel free to email questions or feedback to the BSSw tutorial team at bssw-tutorial@lists.mcs.anl.gov.
If you want to do the hands-on exercises, we’re happy to provide feedback on your pull requests and issues, even after the end of the tutorial.
To find out about future events organized by the IDEAS Productivity Project, you can subscribe to our mailing list (usually ~2 messages/month).
For monthly updates on the Better Scientific Software site, subscribe to our monthly digest.

If you’re interested in this tutorial, you might be interested in these related events taking place at this week’s ECP Annual Meeting or next week’s ECP Community BoF Days.

ECP Annual Meeting

The following links will be accessible only to those who have registered for the ECP Annual Meeting

Day(s)	Time (EDT)	Event
Mon-Fri	12:30pm-1:30pm	IDEAS Desk – informal conversations about developer productivity and software sustainability Room 125 in the Gather.Town Side Meetings area
Tue	1:00pm-3:00pm	Broader Engagement Initiative
Tue	4:00-6:00pm	AD, Hardware and Integration, and Facilities Poster Session Posters on the IDEAS project, BSSw.io resource site, and BSSw Fellowship
Wed	1:00pm-2:00pm	Best Practices #somycodewillseethefuture Breakout
Wed	1:00pm-3:00pm	Panel on Sustainability of ECP Software and Applications
Thu	1:00pm-3:00pm	What Can be Learned from Applying Team of Teams Principles to the ECP Projects PETSc, Trilinos, xSDK, and E4S?

ECP Community BoF Days

The ECP Community BoF Days are open to the public. Registration is required.

Day	Time (EDT)	Event
Wed 11 May	3:00-4:30pm	Fostering Software Sustainability, Productivity and Quality through BSSw.io

Resources from Presentations

Links from the tutorial presentations are listed here for convenience

Module 0: Introduction and Setup
- IDEAS Productivity Project
  - IDEAS Productivity mailing list
- Better Scientific Software site
  - BSSw Digest
  - RSS Feed
- The IDEAS report
- Write to the tutorial authors
- Inclusive Naming Initiative
- Bssw.io resource on Inclusive Naming
Module 1: Motivation and Overview
Module 2: Software Testing Introduction
- Python Build and Test Framework: pyscaffold.org
- Build-Link-Test CMake Framework: llnl-blt.readthedocs.io
- Tutorials for code coverage: Online Tutorial, Another example
- Test drive development example
- CMake Tutorial
- CMake add-test command documentation
- Verification and Validation in Scientific Computing
- Working Effectively with Legacy Code
Module 3: Testing Complex Software
- Useful resources on testing (formerly linked to ~~ideas-productivity.org/resources/howtos/~~)
- Related articles:
  - Methodology for Building Granular Testing in Multicomponent Scientific Software
  - Ongoing verification of a multiphysics community code: FLASH
Module 4: Continuous Integration
- Gitlab CI/CD Concepts
- CI testing examples
  - Flash test
  - Buildtest
  - Nightly
  - ~~www.fluidnumerics.com/resources/cicd-in-the-cloud-for-hpc-applications~~
- Exascale Computing Project CI documentation
- Github Actions
- GitLab CI/CD
- Travis CI service
- Codecov service
- Spack
- Extreme-scale Scientific Software Stack (E4S)
- Best practices for using Docker Hub for CI/CD
Module 5: Summary

Requested Citation

The requested citation the overall tutorial is:

David M. Rogers and Gregory R. Watson, Developing a Testing and Continuous Integration Strategy for your Team tutorial, in Exascale Computing Project Annual Meeting, online, 2022. DOI: 10.6084/m9.figshare.19608927.

Individual modules may be cited as Speaker, Module Title, in Developing a Testing and Continuous Integration Strategy for your Team tutorial…

Acknowledgements

This tutorial is produced by the IDEAS Productivity project.

This work was supported by the U.S. Department of Energy Office of Science, Office of Advanced Scientific Computing Research (ASCR), and by the Exascale Computing Project (17-SC-20-SC), a collaborative effort of the U.S. Department of Energy Office of Science and the National Nuclear Security Administration.