testthat 3.2.0

We’re chuffed to announce the release of testthat 3.2.0. testthat makes it easy to turn your existing informal tests into formal, automated tests that you can rerun quickly and easily. testthat is the most popular unit-testing package for R, and is used by almost 9,000 CRAN and Bioconductor packages. You can learn more about unit testing at https://r-pkgs.org/tests.html.

You can install it from CRAN with:

install.packages("testthat")

testthat 3.2.0 includes relatively few new features but there have been nine patch releases since testthat 3.1.0. These patch releases contained a bunch of experiments that we now believe are ready for the world. So this blog post summarises the changes in 3.1.1, 3.1.2, 3.1.3, 3.1.4, 3.1.5, 3.1.6, 3.1.7, 3.1.8, 3.1.9, and 3.1.10 over the last two years.

Here we’ll focus on the biggest news: new expectations, tweaks to the way that error snapshots are reported, support for mocking, a new way to detect if a test has changed global state, and a bunch of smaller UI improvements.

library(testthat)

Documentation

The first and most important thing to point out is that the second edition of R Packages contains updated and much expanded coverage of testing. Coverage of testing is now split up over three chapters:

• Testing basics
• Designing your test suite
• Advanced testing techniques

There’s also a new vignette about special files ( vignette("special-files")) which describes the various special files that you find in tests/testthat and when you might need to use them.

New expectations

There are a handful of notable new expectations. expect_contains() and expect_in() work similarly to expect_true(all(expected %in% object)) or expect_true(all(object %in% expected)) but give more informative failure messages:

fruits <- c("apple", "banana", "pear")
expect_contains(fruits, "apple")
expect_contains(fruits, "pineapple")
#> Error: `fruits` (`actual`) doesn't fully contain all the values in "pineapple" (`expected`).
#> * Missing from `actual`: "pineapple"
#> * Present in `actual`:   "apple", "banana", "pear"

x <- c(TRUE, FALSE, TRUE, FALSE)
expect_in(x, c(TRUE, FALSE))
x <- c(TRUE, FALSE, TRUE, NA, FALSE)
expect_in(x, c(TRUE, FALSE))
#> Error: `x` (`actual`) isn't fully contained within c(TRUE, FALSE) (`expected`).
#> * Missing from `expected`: NA
#> * Present in `expected`:   TRUE, FALSE

expect_no_error(), expect_no_warning(), and expect_no_message() make it easier (and clearer) to confirm that code runs without errors, warnings, or messages. The default fails if there is any error/warning/message, but you can optionally supply either the message or class arguments to confirm the absence of a specific error/warning/message.

foo <- function(x) {
  if (x < 0) {
    x + "10"
  } else {
    x = 20
  }
}

expect_no_error(foo(-10))
#> Error: Expected `foo(-10)` to run without any errors.
#> ℹ Actually got a <simpleError> with text:
#>   non-numeric argument to binary operator

# No difference here but will lead to a better failure later
# once you've fixed this problem and later introduce a new one
expect_no_error(foo(-10), message = "non-numeric argument")
#> Error: Expected `foo(-10)` to run without any errors matching pattern 'non-numeric argument'.
#> ℹ Actually got a <simpleError> with text:
#>   non-numeric argument to binary operator

Snapshotting changes

expect_snapshot(error = TRUE) has a new display of error messages that strives to be closer to what you see interactively. In particular, you’ll no longer see the error class and you will now see the error call.

• Old display:

  Code
    f()
  Error <simpleError>
    baz
  

• New display:

  Code
    f()
  Condition
    Error in `f()`:
    ! baz
  

If you have used expect_snapshot(error = TRUE) in your package, this means that you will need to re-run and approve your snapshots. We hope this is not too annoying and we believe it is worth it given the more accurate reflection of generated error messages. This will not affect checks on CRAN because, by default, snapshot tests are not run on CRAN.

Mocking

Mocking¹ is a tool for temporarily replacing the implementation of a function in order to make testing easier. Sometimes when testing a function, one part of it is challenging to run in your test environment (maybe it requires human interaction, a live database connection, or maybe it just takes a long time to run). For example, take the following imaginary function. It has a bunch of straightforward computation that would be easy to test but right in the middle of the function it calls complicated() which is hard to test:

my_function <- function(x, y, z) {
  a <- f(x, y)
  b <- g(y, z)
  c <- h(a, b)
  
  d <- complicated(c)
  
  i(d, 1, TRUE)
}

Mocking allows you to temporarily replace complicated() with something simpler, allowing you to test the rest of the function. testthat now supports mocking with local_mocked_bindings(), which temporarily replaces the implementation of a function. For example, to test my_function() you might write something like this:

test_that("my_function() returns expected result", {
  local_mocked_bindings(
    complicated = function(x) TRUE
  )
  ...
})

testthat has a complicated past with mocking. testthat introduced with_mock() in v0.9 (way back in 2014), but we started discovering problems with the implementation in v2.0.0 (2017) leading to its deprecation in v3.0.0 (2020). A few packages arose to fill the gap (like mockery, mockr, and mockthat) but none of their implementations were completely satisfactory. Earlier this year a new approach occurred to me that avoids many of the problems of the previous approaches. This is now implemented in with_mocked_bindings() and local_mocked_bindings(); we’ve been using these new functions for a few months now without problems, and it feels like time to announce to the world.

State inspector

In times gone by it was very easy to accidentally change the state of the world in a test:

test_that("side-by-side diffs work", {
  options(width = 20)
  expect_snapshot(
    waldo::compare(c("X", letters), c(letters, "X"))
  )
})

When you look at a single test it’s easy to spot the problem, and switch to a more appropriate way of temporarily changing the options, like withr::local_options(). But sometimes this mistake crept in a long time ago and is now hiding amongst hundreds or thousands of tests.

In earlier versions of testthat, finding tests that accidentally changed the world was painful: the only way was to painstakingly review each test. Now you can use set_state_inspector() to register a function that’s called before and after every test. If the function returns different values, testthat will let you know. You’ll typically do this either in tests/testhat/setup.R or an existing helper file.

So, for example, to detect if any of your tests have modified options you could use this state inspector:

set_state_inspector(function() {
  list(options = options())
})

Or maybe you’ve seen an R CMD check warning that you’ve forgotten to close a connection:

set_state_inspector(function() {
  list(connections = nrow(showConnections()))
})

And you can of course combine multiple checks just by returning a more complicated list.

UI improvements

testthat 3.2.0 includes a bunch of minor user interface improvements that should make day-to-day use of testthat more enjoyable. Some of our favourite highlights are:

• Parallel testing now works much better with snapshot tests. (And updates to the processx package means that testthat no longer leaves processes around if you terminate a test process early.)
• We use an improved algorithm to find the source reference associated with an expectation/error/warning/skip. We now look for the most recent call (within inside test_that() that has known source. This generally gives more specific locations than the previous approach and gives much better locations if an error occurs in an exit handler.
• Tracebacks are no longer truncated and we use rlang’s default tree display; this should make it easier to track down problems when testing in non-interactive contexts.
• Assuming you have a recent RStudio, test failures are now clickable, taking you to the line where the problem occurred. Similarly, when a snapshot test changes, you can now click that suggested code to run the appropriate snapshot_accept() call.
• Skips are now only shown at the end of reporter summaries, not as tests are run. This makes them less intrusive in interactive tests while still allowing you to verify that the correct tests are skipped.

Acknowledgements

A big thanks to all 127 contributors who helped make these last 10 release of testthat happen, whether it be through contributed code or filing issues: @ALanguillaume, @alessandroaccettulli, @ambica-aas, @annweideman, @aronatkins, @ashander, @AshesITR, @astayleraz, @ateucher, @avraam-inside, @b-steve, @bersbersbers, @billdenney, @Bisaloo, @cboettig, @cderv, @chendaniely, @ChrisBeeley, @ColinFay, @CorradoLanera, @daattali, @damianooldoni, @DanChaltiel, @danielinteractive, @DavisVaughan, @daynefiler, @dbdimitrov, @dcaseykc, @dgkf, @dhicks, @dimfalk, @dougwyu, @dpprdan, @dvg-p4, @elong0527, @Enchufa2, @etiennebacher, @FlippieCoetser, @florisvdh, @gaborcsardi, @gareth-j, @gavinsimpson, @ghill-fusion, @hadley, @heavywatal, @hfrick, @hhau, @hpages, @hsloot, @hughjonesd, @IndrajeetPatil, @jameslairdsmith, @jamieRowen, @jayruffell, @JBGruber, @jennybc, @JohnCoene, @jonathanvoelkle, @jonthegeek, @josherrickson, @kalaschnik, @kapsner, @kevinushey, @kjytay, @krivit, @krlmlr, @larmarange, @lionel-, @llrs, @luma-sb, @machow, @maciekbanas, @maelle, @majr-red, @maksymiuks, @mardam, @MarkMc1089, @markschat, @MatthieuStigler, @maurolepore, @maxheld83, @mbojan, @mcol, @mgirlich, @MichaelChirico, @mkb13, @mkoohafkan, @MKyhos, @moodymudskipper, @Mosk915, @mpjashby, @ms609, @mtmorgan, @musvaage, @nealrichardson, @netique, @njtierney, @olivroy, @osorensen, @pbulsink, @peterdesmet, @r2evans, @radbasa, @remlapmot, @rfineman, @rgayler, @romainfrancois, @s-fleck, @salim-b, @schloerke, @sorhawell, @StatisMike, @StatsMan53, @stela2502, @stla, @t-kalinowski, @tansaku, @tomliptrot, @torres-pedro, @wes-brooks, @wfmueller29, @wleoncio, @wurli, @yogat3ch, @yuliaUU, @yutannihilation, and @zsigmas.