Julia Language Writing a Test Set


Example

0.5.0

In version v0.5, test sets are built into the standard library Base.Test module, and you don't have to do anything special (besides using Base.Test) to use them.

0.4.0

Test sets are not part of Julia v0.4's Base.Test library. Instead, you have to REQUIRE the BaseTestNext module, and add using BaseTestNext to your file. To support both version 0.4 and 0.5, you could use

if VERSION ≥ v"0.5.0-dev+7720"
    using Base.Test
else
    using BaseTestNext
    const Test = BaseTestNext
end

It is helpful to group related @tests together in a test set. In addition to clearer test organization, test sets offer better output and more customizability.

To define a test set, simply wrap any number of @tests with a @testset block:

@testset "+" begin
    @test 1 + 1 == 2
    @test 2 + 2 == 4
end

@testset "*" begin
    @test 1 * 1 == 1
    @test 2 * 2 == 4
end

Running these test sets prints the following output:

Test Summary: | Pass  Total
  +           |    2      2

Test Summary: | Pass  Total
  *           |    2      2

Even if a test set contains a failing test, the entire test set will be run to completion, and the failures will be recorded and reported:

@testset "-" begin
    @test 1 - 1 == 0
    @test 2 - 2 == 1
    @test 3 - () == 3
    @test 4 - 4 == 0
end

Running this test set results in

-: Test Failed
  Expression: 2 - 2 == 1
   Evaluated: 0 == 1
 in record(::Base.Test.DefaultTestSet, ::Base.Test.Fail) at ./test.jl:428
    ...
-: Error During Test
  Test threw an exception of type MethodError
  Expression: 3 - () == 3
  MethodError: no method matching -(::Int64, ::Tuple{})
    ...
Test Summary: | Pass  Fail  Error  Total
  -           |    2     1      1      4
ERROR: Some tests did not pass: 2 passed, 1 failed, 1 errored, 0 broken.
    ...

Test sets can be nested, allowing for arbitrarily deep organization

@testset "Int" begin
    @testset "+" begin
        @test 1 + 1 == 2
        @test 2 + 2 == 4
    end
    @testset "-" begin
        @test 1 - 1 == 0
    end
end

If the tests pass, then this will only show the results for the outermost test set:

Test Summary: | Pass  Total
  Int         |    3      3

But if the tests fail, then a drill-down into the exact test set and test causing the failure is reported.

The @testset macro can be used with a for loop to create many test sets at once:

@testset for i in 1:5
    @test 2i == i + i
    @test i^2 == i * i
    @test i ÷ i == 1
end

which reports

Test Summary: | Pass  Total
  i = 1       |    3      3
Test Summary: | Pass  Total
  i = 2       |    3      3
Test Summary: | Pass  Total
  i = 3       |    3      3
Test Summary: | Pass  Total
  i = 4       |    3      3
Test Summary: | Pass  Total
  i = 5       |    3      3

A common structure is to have outer test sets test components or types. Within these outer test sets, inner test sets test behaviour. For instance, suppose we created a type UniversalSet with a singleton instance that contains everything. Before we even implement the type, we can use test-driven development principles and implement the tests:

@testset "UniversalSet" begin
    U = UniversalSet.instance
    @testset "egal/equal" begin
        @test U === U
        @test U == U
    end

    @testset "in" begin
        @test 1 in U
        @test "Hello World" in U
        @test Int in U
        @test U in U
    end

    @testset "subset" begin
        @test Set() ⊆ U
        @test Set(["Hello World"]) ⊆ U
        @test Set(1:10) ⊆ U
        @test Set([:a, 2.0, "w", Set()]) ⊆ U
        @test U ⊆ U
    end
end

We can then start implementing our functionality until it passes our tests. The first step is to define the type:

immutable UniversalSet <: Base.AbstractSet end

Only two of our tests pass right now. We can implement in:

immutable UniversalSet <: Base.AbstractSet end
Base.in(x, ::UniversalSet) = true

This also makes some of our subset tests to pass. However, the issubset () fallback doesn't work for UniversalSet, because the fallback tries to iterate over elements, which we can't do. We can simply define a specialization that makes issubset return true for any set:

immutable UniversalSet <: Base.AbstractSet end
Base.in(x, ::UniversalSet) = true
Base.issubset(x::Base.AbstractSet, ::UniversalSet) = true

And now, all our tests pass!