aegis -Test(1) aegis -Test(1) NNAAMMEE aegis -Test - run tests SSYYNNOOPPSSIISS aaeeggiiss --TTeesstt [ _f_i_l_e_-_n_a_m_e... ][ _o_p_t_i_o_n... ] aaeeggiiss --TTeesstt --IINNDDeeppeennddeenntt [ _f_i_l_e_-_n_a_m_e... ][ _o_p_t_i_o_n... ] aaeeggiiss --TTeesstt --LLiisstt [ _o_p_t_i_o_n... ] aaeeggiiss --TTeesstt --HHeellpp DDEESSCCRRIIPPTTIIOONN The _a_e_g_i_s _-_T_e_s_t command is used to run tests. If no files are named, all relevant tests are run. By default both automatic and manual tests are run. You may name directories on the command line, and all relevant tests in that directory tree in the change will be run. It is an error if there are no relevant tests. Each architecture must be tested separately. This is because there may be subtle problems that are only revealed on some architectures. Some projects may also have different code for different architectures. The status of the last test run is remembered so that tests are not run if there is no need. (This does not apply to _-_R_E_G_r_e_s_s_i_o_n tests, unfortunately.) Tests must be re-run if the test previously failed, if the test file has changed, if there has been a build, and for each architecture. FFiillee NNaammee IInntteerrpprreettaattiioonn The aegis program will attempt to determine the project file names from the file names given on the command line. All file names are stored within aegis projects as relative to the root of the baseline directory tree. The development directory and the integration directory are shadows of this baseline directory, and so these relative names apply here, too. Files named on the command line are first converted to absolute paths if necessary. They are then compared with the baseline path, the development directory path, and the integration directory path, to determine a baseline-relative name. It is an error if the file named is outside one of these directory trees. The --BBAAssee__RREEllaattiivvee option may be used to cause relative filenames to be interpreted as relative to the baseline path; absolute filenames will still be compared with the various paths in order to determine a baseline-relative name. The _r_e_l_a_t_i_v_e___f_i_l_e_n_a_m_e___p_r_e_f_e_r_e_n_c_e in the user configuration file may be used to modify this default behavior. See _a_e_u_c_o_n_f(5) for more information. TTEESSTT PPRROOCCEESSSS Each change is required to be accompanied by tests, and those tests are required to be run against the built development directory, and they must pass. This ensures that new functionality is accompanied by tests to verify its correctness, and bug fixes are accompanied by tests which confirm that the bug has been fixed. RReeggrreessssiioonn TTeessttss Tests are treated as any other source file, and are maintained in the baseline and history with all other source files. The tests which must accompany every change accumulate in the project baseline, providing a definition of correct function for the baseline. These accumulated tests may be executed using an ``aegis -REGression'' command, to verify that the project will not ``regress'' as a result of a change. BBaasseelliinnee TTeessttss Bug fixes are required to have their tests _f_a_i_l against the project baseline (in contrast to the development directory). This ensures that the test actually demonstrates the bug in the baseline, as well as demonstrating that it is fixed by the change. New functionality trivially fails against the baseline, and so aegis does not attempt to guess if a test is a bug fix test or new functionality test, it simply requires tests to fail against the baseline. This requirement applies both to new tests being created by a change and also to tests which have been copied into a change for modification. RReevviieewwiinngg TTeessttss Reviewers may be confident that aegis has enforced the test requirements; that a change must have tests, that the change must build, that the tests pass against the development directory, and that the tests fail against the baseline. These conditions are enforced by _a_e_d_e(1) and the change will not be advanced to the _b_e_i_n_g _r_e_v_i_e_w_e_d state until these conditions are met. Reviewers should thus review tests for _c_o_m_p_l_e_t_e_n_e_s_s of coverage of the code in the change, and insensitivity to changes in the execution environment (e.g. not date sensitive). Reviewers should also use ``aegis -list change_details'' to verify that a change does or does not have testing exemptions. EExxeemmppttiioonnss Various test exemptions may be granted by project administrators, see _a_e_p_a(1) and _a_e_p_a_t_t_r(5) for more information. Copying tests into a change, or adding new tests to a change, may cancel those exemptions. TTEESSTT CCOOMMMMAANNDD CCOONNFFIIGGUURRAATTIIOONN The command used to execute tests is defined by the _t_e_s_t___c_o_m_m_a_n_d field in the project _c_o_n_f_i_g file (see _a_e_p_c_o_n_f(5) for more information), this defaults to using the Bourne shell if not set. The current directory will be the top of the appropriate directory tree. If tests require temporary files, they should create them in _/_t_m_p, as a test cannot expect to have write permission in the current directory. If you want to use a more sophisticated test engine, rather than a simple shell script, but this test engine does not return result codes suitable for use with aegis, you could wrap it in a shell script which re-writes the exit status into the values aegis expects. You could also achieve the same results by writing a more complex _t_e_s_t___c_o_m_m_a_n_d in the project _c_o_n_f_i_g file. It is also possible to write test commands which are able to test more than one file at once. This is controlled by the _b_a_t_c_h___t_e_s_t___c_o_m_m_a_n_d field of the project _c_o_n_f_i_g file. In this case, the ${output} substitution indicates the name of a file the test command must create, in _a_e_t_e_s_t(5) format, to contain the results of the tests run. This is often used on systems with multiple CPUs or the ability to distribute jobs across several computers on a network. SSuubbssttiittuuttiioonnss All of the _a_e_s_u_b(5) substitutions are available in the test commands. Some of them are of particular note: _A_R_C_H_i_t_e_c_t_u_r_e This substitution is replaced by the name of the architecture to be tested. _S_e_a_r_c_h___P_a_t_h This substitution is replaced by a colon separated list of absolute paths to search when looking for test support files. _S_e_a_r_c_h___P_a_t_h___E_x_e_c_u_t_a_b_l_e This substitution is replaced by a colon separated list of absolute paths to search when looking for executable support files (library files and sub- commands). Most of the time _$_S_e_a_r_c_h___P_a_t_h___E_x_e_c_u_t_a_b_l_e are exactly the same. However, during ``aegis -t -bl'' they will be different, with _$_S_e_a_c_h___P_a_t_h starting at the development directory (the test being run) and _$_S_e_a_c_h___P_a_t_h___E_x_e_c_u_t_a_b_l_e starting at the baseline (the executable being run). TTeesstt RReessuulltt CCooddeess As each test is run (via the _t_e_s_t___c_o_m_m_a_n_d field in the project _c_o_n_f_i_g file), aegis determines whether the test succeeded or failed by looking at its exit status. This exit status is mostly as expected for UNIX commands. Success A test should exit 0 to indicate success, i.e. that the specific function under test worked as expected. Failure A test should exit 1 to indicate failure, i.e. that the specific function under test did not work as expected. No Result A test should exit 2 to indicate no result, i.e. that the specific function under test could not be exersized because something else went wrong. For example, running out of disk space when creating the test input files in the _/_t_m_p directory. Actually, any exit code other than 0 or 1 will be interpreted as ``no result''. However, always using 0, 1 or 2 means that if a new result code is required by a later release of aegis, your existing tests will continue to work. TTEESSTT CCOORRRREELLAATTIIOONNSS The ``aegis -Test -SUGgest'' command may be used to have aegis suggest suitable regression tests for your change, based on the source files in your change. This automatically focuses testing effort to relevant tests, reducing the number of regression tests necessary to be confident that you have not introduced a bug. The test correlations are generated by the ``aegis -Integrate_Pass'' command, which associates each test in the change with each source file in the change. Thus, each source file accumulates a list of tests which have been associated with it in the past. This is not as exact as code coverage analysis, but is a reasonable approximation in practice. The _a_e_c_p(1) and _a_e_n_f(1) commands are used to associate files with a change. While they do not actively perform the association, these are the files used by _a_e_i_p_a_s_s(1) and _a_e_t(1) to determine which source files are associated with which tests. TTeesstt CCoorrrreellaattiioonn AAccccuurraaccyy Assuming that the testing correlations are accurate and that the tests are evenly distributed across the function space, there will be a less than _1_/_n_u_m_b_e_r chance that a relevant test has not been run by the ``aegis -Test -SUGgest _n_u_m_b_e_r'' command. A small amount of noise is added to the test weighting, so that unexpected things are sometimes tested, and the same tests are not run every time. Test correlation accuracy can be improved by ensuring that: +o Each change should be strongly focused, with no gratuitous file inclusions. This avoids spurious correlations. +o Each item of new functionality should be added in an individual change, rather than several together. This strongly correlates tests with functionality. +o Each bug should be fixed in an individual change, rather than several together. This strongly correlates tests with functionality. +o Test correlations will be lost if files are moved. This is because correlations are by name. The best way for tests to correlate accurately with source files is when a change contains a test and exactly those files relating to the functionality under test. Too many spurious files will weaken the usefulness of the testing correlations. OOPPTTIIOONNSS The following options are understood: --AAUUTTOOmmaattiicc This option may be used to specify automatic tests. Automatic tests require no human assistance. --BBaasseeLLiinnee This option may be used to specify that the project baseline is the subject of the command. --BBAAssee__RREEllaattiivvee This option may be used to cause relative filenames to be considered relative to the base of the source tree. See _a_e_u_c_o_n_f(5) for the corresponding user preference. --CCUUrrrreenntt__RREEllaattiivvee This option may be used to cause relative filenames to be considered relative to the current directory. This is usually the default. See _a_e_u_c_o_n_f(5) for the corresponding user preference. --CChhaannggee _n_u_m_b_e_r This option may be used to specify a particular change within a project. When no --CChhaannggee option is specified, the _A_E_G_I_S___C_H_A_N_G_E environment variable is consulted. If that does not exist, the user's _$_H_O_M_E_/_._a_e_g_i_s_r_c file is examined for a default change field (see _a_e_u_c_o_n_f(5) for more information). If that does not exist, when the user is only working on one change within a project, that is the default change number. Otherwise, it is an error. --FFOOrrccee This option may be used to specify that all tests should be run, even if the status of the last test run indicates that there is no need to run a specific test. --HHeellpp This option may be used to obtain more information about how to use the _a_e_g_i_s program. --IINNDDeeppeennddeenntt This option is used to specify that the test is to be run independent of any particular change. If no tests are named, all tests in the baseline will be run. --LLiisstt This option may be used to obtain a list of suitable subjects for this command. The list may be more general than expected. --MMAANNuuaall This option may be used to specify manual tests. Manual tests require some human intervention, e.g.: confirmation of some screen behavior (X11, for instance), or some user action, "unplug ethernet cable now". --NNoott__LLooggggiinngg This option may be used to disable the automatic logging of output and errors to a file. This is often useful when several aegis commands are combined in a shell script. --PPEErrsseevveerree This option may be used to specify that all tests should be run, even if some fail. Defaults to the user's _p_e_r_s_e_v_e_r_e___p_r_e_f_e_r_e_n_c_e if not specified, see _a_e_u_c_o_n_f(5) for more information. --NNoo__PPEErrsseevveerree This option may be used to specify that the test run should stop after the first failure. Defaults to the user's _p_e_r_s_e_v_e_r_e___p_r_e_f_e_r_e_n_c_e if not specified, see _a_e_u_c_o_n_f(5) for more information. --PPrroojjeecctt _n_a_m_e This option may be used to select the project of interest. When no --PPrroojjeecctt option is specified, the _A_E_G_I_S___P_R_O_J_E_C_T environment variable is consulted. If that does not exist, the user's _$_H_O_M_E_/_._a_e_g_i_s_r_c file is examined for a default project field (see _a_e_u_c_o_n_f(5) for more information). If that does not exist, when the user is only working on changes within a single project, the project name defaults to that project. Otherwise, it is an error. --RREEGGrreessssiioonn This option is used to specify that the regression test suite is to be run. The regression test suite consists of all tests in the baseline which do not appear in the change. It is an error if there are no regression tests. You may not name tests on the command line when using the -REGression option. You may name individual tests to be run on the command line, without using the -REGression option; if they are not part of the change, the tests of the same name in the baseline will be run. --SSUUGGggeesstt [ _n_u_m_b_e_r ] The ``_a_e_g_i_s _-_I_n_t_e_g_r_a_t_e___P_a_s_s'' command collects test correlation statistics when changes are integrated. This option may be used to request that aegis suggest which tests should be run, using these testing correlations. If no number is specified, 10 tests will be suggested. This option implies the --RREEGGrreessssiioonn option. --SSUUGGggeesstt__NNooiissee _n_u_m_b_e_r This option may be used to control the amount of noise injected into the test selection performed by the --SSUUGGggeesstt option. The number is a percentage of noise to be injected. Defaults to 10 if not specified. The injection of noise ensures that a variety of tests are run on subsequent runs, and also some from left-field as a sanity check. --TTEERRssee This option may be used to cause listings to produce the bare minimum of information. It is usually useful for shell scripts. --VVeerrbboossee This option may be used to cause aegis to produce more output. By default aegis only produces output on errors. When used with the --LLiisstt option this option causes column headings to be added. --WWaaiitt This option may be used to require Aegis commands to wait for access locks, if they cannot be obtained immediately. Defaults to the user's _l_o_c_k___w_a_i_t___p_r_e_f_e_r_e_n_c_e if not specified, see _a_e_u_c_o_n_f(5) for more information. --NNoo__WWaaiitt This option may be used to require Aegis commands to emit a fatal error if access locks cannot be obtained immediately. Defaults to the user's _l_o_c_k___w_a_i_t___p_r_e_f_e_r_e_n_c_e if not specified, see _a_e_u_c_o_n_f(5) for more information. See also _a_e_g_i_s(1) for options common to all aegis commands. All options may be abbreviated; the abbreviation is documented as the upper case letters, all lower case letters and underscores (_) are optional. You must use consecutive sequences of optional letters. All options are case insensitive, you may type them in upper case or lower case or a combination of both, case is not important. For example: the arguments "-project, "-PROJ" and "-p" are all interpreted to mean the --PPrroojjeecctt option. The argument "-prj" will not be understood, because consecutive optional characters were not supplied. Options and other command line arguments may be mixed arbitrarily on the command line, after the function selectors. The GNU long option names are understood. Since all option names for _a_e_g_i_s are long, this means ignoring the extra leading '-'. The "----_o_p_t_i_o_n==_v_a_l_u_e" convention is also understood. RREECCOOMMMMEENNDDEEDD AALLIIAASS The recommended alias for this command is csh% alias aet 'aegis -t \!* -v' sh$ aet(){aegis -t "$@" -v} EERRRROORRSS It is an error if the change is not in one of the _b_e_i_n_g _d_e_v_e_l_o_p_e_d or _b_e_i_n_g _i_n_t_e_g_r_a_t_e_d states. It is an error if the change is not assigned to the current user. It is an error if your have no relevant tests and no relevant exemption. EEXXIITT SSTTAATTUUSS The _a_e_g_i_s command will exit with a status of 1 on any error. The _a_e_g_i_s command will only exit with a status of 0 if there are no errors. EENNVVIIRROONNMMEENNTT VVAARRIIAABBLLEESS See _a_e_g_i_s(1) for a list of environment variables which may affect this command. SSEEEE AALLSSOO _a_e_b(1) build a change _a_e_c_a(1) modify the attributes of a change _a_e_d_b(1) begin development of a change _a_e_i_b(1) begin integration of a change _a_e_n_t(1) add a new test to a change _a_e_c_p(1) copy an existing test into a change _a_e_p_c_o_n_f(5) project configuration file format _a_e_u_c_o_n_f(5) user configuration file format CCOOPPYYRRIIGGHHTT aegis version .C001 Copyright (C) 1991, 1992, 1993, 1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001 Peter Miller; All rights reserved. The aegis program comes with ABSOLUTELY NO WARRANTY; for details use the '_a_e_g_i_s _-_V_E_R_S_i_o_n _L_i_c_e_n_s_e' command. This is free software and you are welcome to redistribute it under certain conditions; for details use the '_a_e_g_i_s _-_V_E_R_S_i_o_n _L_i_c_e_n_s_e' command. AAUUTTHHOORR Peter Miller E-Mail: millerp@canb.auug.org.au /\/\* WWW: http://www.canb.auug.org.au/~millerp/ Reference Manual Aegis 1