Running autofonce

Project Initialization

Before running tests, you will need to configure the project to run autofonce. Some projects may come (in the future) with a predefined configuration, with the presence of a .autofonce file. Otherwise, you will need to create a file autofonce.toml.

If you need to create this autofonce.toml, you can use:

$ autofonce init

Using this command makes autofonce try to autodetect the project, in which case it can generate a good example. Autodetection will only work for known projects, currently only gnucobol. For autodetection to work, the name of a directory in the path should match the name of the known project.

Once this file has been generated, you should check its content to verify that it matches your own setting. See the Project Configuration section for more information.

In the presence of a autofonce.toml file, autofonce will run the tests from the directory containing this file, within a sub-directory _autofonce/.

If autofonce.toml is absent and the .autofonce project file is used, then autofonce will use the build directory detected by the build_anchors as the directory where to create the _autofonce/ directory.

Listing all tests

Once you have a working project configuration, you can try to read the testsuite using:

$ autofonce list

This command should display the list of tests it was able to read, or errors if it could not read the testsuite.

Running all tests

Once you are able to read the testsuite, you can run the tests with:

$ autofonce run

If a failure appends, you can check the directory _autofonce/NNNN/ where NNNN is the test numeric identifier.

For skipped test, you can check the file _autofonce/results.log.

Unless you use the arguments --keep-more or --keep-all, test sub-directories are not kept if the tests are skipped, or if they succeed.

Within a test sub-directory, you should find:

  • env_autofonce.sh: the script used to initialize the environment for the test. It contains the content of the file autofonce.env. If you want to call commands in this directory with the same env as the tests, you should use:

    . ./env_autofonce.sh

  • 00_CHECK.sh: the shell script to run the first check of the test

  • 00_CHECK.out: the stdout output of the test

  • 00_CHECK.err: the stderr output of the test

  • 00_CHECK.???.expected: if the comparison between the outputs failed, this file contains what was expected by the test

  • CHECK files are numbered, with nested numbering for nested checks

  • other files are either files created by AT_DATA macros, or files generated by checks

Selecting Tests

When you are working on tests, you sometimes want to select only a few of them, instead of running the whole testsuite that will take a lot of time.

You can use the following options to select tests, or a combination of them:

  • --before N: run selected tests before N included

  • --after N: run selected tests after N included

  • --ids 'N1 N2 N3': run tests N1, N2 and N3

  • -k KEYWORD: run tests matching KEYWORD, as specified by the AT_KEYWORD(...) macro

  • --not 'K1 K2': do not run selected tests that match keywords K1 or K2

  • --failed: run only selected tests that failed during previous run

Promoting tests

Sometimes, you change the behavior of your tool, and some tests need to be updated to match the new behavior. Such a work can be cumbersome, especially when the expected stdout/stderr of the tests are specified.

autofonce can help you be promoting the outputs of failed tests directly inside the corresponding tests files.

Here is the processus:

  • Run the tests:

    autofonce run

  • Check all the failed tests and verify that their reasons for failing is ok. A way to do that is to use this command:

    autofonce promote

    The command will display a diff of what it would do to patch the tests, i.e. show the difference between the output and what was expected.

  • Once you think the new output of the tests is ok, you can ask autofonce to update the files:

    autofonce promote --apply

  • It is still a good idea to check the results, for example using git diff or a similar tool.

Note that autofonce will add a comment # promoted on <DATE> to every updated test. You may remove this comment using the option --no-comment.

Working with multiple testsuites

If your project contains several testsuites, or if you want to be able to run the same testsuites with different settings (for example, before installation and after installation), you may define several such testsuites in autofonce.toml or .autofonce.

You can then use the -t TESTSUITE option to run a different testsuite than the default one (i.e. the first defined one):

autofonce run -t installed

See Testsuite Descriptions for more information.