baselining

Baselining and re-baselining tests

Many tests have a file or files that are used as the "correct" answer. Too large a deviation from this baseline file is considered a "diff". Many times the application will change and while not wrong, will cause a difference in the results.

This happens frequent enough that vvtest has a mechanism to facilitate re-baselining.

Simple baseline file copy

Consider the following rebase.vvt file:

#VVT: link : baseline.txt
#VVT: baseline : output.txt, baseline.txt

import script_util as util

with open( 'output.txt', 'wt' ) as fp:
  fp.write( 'Answer = 1.2\n' )

if util.unixdiff( 'output.txt', 'baseline.txt' ):
  util.exit_diff()

This test produces an "answer" in the form of a file called "output.txt". It then compares this output to a baseline file, "baseline.txt".

Now suppose the answer changes to "1.3" and the baseline file needs to be updated. This can be accomplished by running the test (which should diff), then running vvtest -b. For example:

$ touch baseline.txt
$ vvtest -w
...
==================================================
diff        0:01 02/18 09:16:59 TestResults.ceelan/rebase
==================================================
Summary:
    completed: 1
          1 diff
    total: 1

$ vvtest -b
==================================================
diff        0:01 02/18 09:16:59 TestResults.ceelan/rebase
==================================================
Test list:
    completed: 1
          1 diff
    total: 1

baselining rebase...done

This copied the output.txt file in the execution directory over the top of baseline.txt in the test source directory. If the test is run again, it should pass:

$ vvtest -R
==================================================
pass        0:01 02/18 09:20:24 TestResults.ceelan/rebase
==================================================
Summary:
    completed: 1
          1 pass
    total: 1

Controlling the tests that get rebaselined

By default, only (and all) tests that previously resulted in a "diff" are rebaselined.

However, filtering can be added to the "vvtest -b" command line. For example, "vvtest -b -p np=1" would only rebaseline tests with a "diff" and whose parameter "np" is one. The command "vvtest -b -k foobar" would rebaseline tests with a "diff" and whose keywords include "foobar".

To rebaseline tests with a different result than "diff", just include one or more result keywords in the -k filter. For example, "vvtest -b -k diff/pass/fail" would rebaseline all tests that passed, diff-ed or failed. The command "vvtest -b -k fail" would only rebaseline tests that failed.

Rebaseline syntax

There are a few ways to do rebaselining. The first is to use basic file copy:

#VVT: baseline (attributes) : testfile, baseline [testfile2, baseline2 [...] ]

Note that standard attributes can be used here. For example,

#VVT: baseline (parameters="np=1") : testfile.txt, baseline.txt

Expressions can be used, such as

#VVT: baseline (parameters="np=1 or np=2") : testfile.txt, baseline.txt

Multiple files can be rebaselined such as

#VVT: baseline : testfile1.txt,baseline1.txt testfile2.txt,baseline2.txt

Or in line continuation form:

#VVT: baseline : testfile1.txt, baseline1.txt
#VVT::           testfile2.txt, baseline2.txt

A different way to do rebaselining is with a script. A script can be called that will be executed when vvtest -b is run, which must perform the file copying. The syntax is

#VVT: baseline : some_script.py

Instead of a separate script, the test script itself (e.g., the "rebase.vvt" file in the first example on this page) can be used to do the rebaselining. Specify the command line argument to be sent into the test script like this:

#VVT: baseline : --baseline

In this case, the test file will be run with command line option --baseline. The option name specified here is arbitrary, but must start with a hyphen. In fact, vvtest distinguishes between the rebaseline methods as follows:

1. If the value has one or more commas in it, then the file copy method is performed
2. If the value starts with a hyphen, then a command line option to the *.vvt file is assumed
3. Else an external script is assumed