Running :code:`autofonce`
=========================

Project Initialization
----------------------

Before running tests, you will need to configure the project to run
:code:`autofonce`. Some projects may come (in the future) with a
predefined configuration, with the presence of a :code:`.autofonce`
file. Otherwise, you will need to create a file :code:`autofonce.toml`.

If you need to create this :code:`autofonce.toml`, you can use::

  $ autofonce init

Using this command makes :code:`autofonce` try to autodetect the
project, in which case it can generate a good example. Autodetection
will only work for *known* projects, currently only
:code:`gnucobol`. For autodetection to work, the name of a directory
in the path should match the name of the known project.

Once this file has been generated, you should check its content to
verify that it matches your own setting. See the
:any:`Project Configuration` section for more information.

In the presence of a :code:`autofonce.toml` file, :code:`autofonce`
will run the tests from the directory containing this file, within a
sub-directory :code:`_autofonce/`.

If :code:`autofonce.toml` is absent and the :code:`.autofonce` project
file is used, then :code:`autofonce` will use the build directory
detected by the :code:`build_anchors` as the directory where to create
the :code:`_autofonce/` directory.

Listing all tests
-----------------

Once you have a working project configuration, you can try to read the
testsuite using::

  $ autofonce list

This command should display the list of tests it was able to read, or
errors if it could not read the testsuite.

Running all tests
-----------------

Once you are able to read the testsuite, you can run the tests with::

  $ autofonce run

If a failure appends, you can check the directory
:code:`_autofonce/NNNN/` where :code:`NNNN` is the test numeric
identifier.

For skipped test, you can check the file :code:`_autofonce/results.log`.

Unless you use the arguments :code:`--keep-more` or :code:`--keep-all`,
test sub-directories are not kept if the tests are skipped, or if they
succeed.

Within a test sub-directory, you should find:

* :code:`env_autofonce.sh`: the script used to initialize the environment
  for the test. It contains the content of the file :code:`autofonce.env`.
  If you want to call commands in this directory with the same env as the
  tests, you should use::

    . ./env_autofonce.sh

* :code:`00_CHECK.sh`: the shell script to run the first check of the test
* :code:`00_CHECK.out`: the stdout output of the test
* :code:`00_CHECK.err`: the stderr output of the test
* :code:`00_CHECK.???.expected`: if the comparison between the outputs
  failed, this file contains what was expected by the test
* :code:`CHECK` files are numbered, with nested numbering for nested checks
* other files are either files created by :code:`AT_DATA` macros, or files
  generated by checks

Selecting Tests
---------------

When you are working on tests, you sometimes want to select only a few
of them, instead of running the whole testsuite that will take a lot
of time.

You can use the following options to select tests, or a combination of
them:

* :code:`--before N`: run selected tests before :code:`N` included
* :code:`--after N`: run selected tests after :code:`N` included
* :code:`--ids 'N1 N2 N3'`: run tests :code:`N1`, :code:`N2` and :code:`N3`
* :code:`-k KEYWORD`: run tests matching :code:`KEYWORD`, as specified by
  the :code:`AT_KEYWORD(...)` macro
* :code:`--not 'K1 K2'`: do not run selected tests that match keywords
  :code:`K1` or :code:`K2`
* :code:`--failed`: run only selected tests that failed during previous run

Promoting tests
---------------

Sometimes, you change the behavior of your tool, and some tests need
to be updated to match the new behavior. Such a work can be
cumbersome, especially when the expected stdout/stderr of the tests
are specified.

:code:`autofonce` can help you be promoting the outputs of failed
tests directly inside the corresponding tests files.

Here is the processus:

* Run the tests::

    autofonce run

* Check all the failed tests and verify that their reasons for failing is
  ok. A way to do that is to use this command::

    autofonce promote

  The command will display a diff of what it would do to *patch* the
  tests, i.e. show the difference between the output and what was
  expected.

* Once you think the new output of the tests is ok, you can ask
  :code:`autofonce` to update the files::

     autofonce promote --apply

* It is still a good idea to check the results, for example using
  :code:`git diff` or a similar tool.

Note that :code:`autofonce` will add a comment :code:`# promoted on
<DATE>` to every updated test. You may remove this comment using the
option :code:`--no-comment`.


Working with multiple testsuites
--------------------------------

If your project contains several testsuites, or if you want to be able
to run the same testsuites with different settings (for example,
before installation and after installation), you may define several
such testsuites in :code:`autofonce.toml` or :code:`.autofonce`.

You can then use the :code:`-t TESTSUITE` option to run a different
testsuite than the default one (i.e. the first defined one)::

  autofonce run -t installed

See :any:`Testsuite Descriptions` for more information.