.\"
.\"     aegis - project change supervisor
.\"     Copyright (C) 1999-2002, 2006-2008, 2010, 2012 Peter Miller
.\"
.\"     This program is free software; you can redistribute it and/or modify
.\"     it under the terms of the GNU General Public License as published by
.\"     the Free Software Foundation; either version 3 of the License, or
.\"     (at your option) any later version.
.\"
.\"     This program is distributed in the hope that it will be useful,
.\"     but WITHOUT ANY WARRANTY; without even the implied warranty of
.\"     MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
.\"     GNU General Public License for more details.
.\"
.\"     You should have received a copy of the GNU General Public License
.\"     along with this program. If not, see
.\"     <http://www.gnu.org/licenses/>.
.\"
.nh 2 "Why Bother?"
.LP
Writing tests is extra work, compared to the way many small (and some
not\[hy]so\[hy]small) software shops operate.
For this reason, the testing requirement may  be turned off.
.LP
The win is that the tests hang around forever,
catching minor and major slips before they become
embarrassing "features" in a released product.
Prevention is cheaper than cure in this case,
the tests save work down the track.
.LP
All of the "extra work" of writing tests is a long\[hy]term win,
where old problems never again reappear.
All of the "extra work" of reviewing changes means that another pair of eyes
sees the code and finds potential problems before they manifest
themselves in shipped product.
All of the "extra work" of integration ensures that the baseline always works,
and is always self\[hy]consistent.
All of the "extra work" of having a baseline and separate
development directories allows multiple parallel development,
with no inter\[hy]developer interference;
and the baseline always works, it is never in an "in\[hy]between" state.
In each case,
not doing this "extra work" is a false economy.
.LP
The existence of these tests, though,
is what determines which projects are most suited to Aegis and which are not.
It should be noted that suitability is a continuous scale,
not black\[hy]and\[hy]white.
With effort and resources,
almost anything fits.
.nh 3 "Projects for which Aegis' Testing is Most Suitable"
.LP
Projects most suited to supervision by Aegis are straight programs.
What the non\[hy]systems\[hy]programmers out there call "tools" and
sometimes "applications".
These are programs which take a pile of input,
chew on it,
and emit a pile of output.
The tests can then compare actual outputs with expected outputs.
.LP
As an example,
you could be writing a
.I sed (1)
look\[hy]alike,
a public domain clone of the
.UX
sed utility.
You could write tests which exercise every feature (insertion, deletion, etc.)
and generate the expected output with the real
.UX
sed.
You write the code,
and run the tests;
you can immediately see if the output matches expectations.
.LP
This is a simple example.
More complex examples exist,
such as Aegis itself.
The Aegis program is used to supervise its own development.
Tests consist of sequences of commands and expected results
are tested for.
.LP
Other types of software have been developed using Aegis:
compilers and interpreters,
client\[hy]server model software,
magnetic tape utilities,
graphics software such as a ray\[hy]tracer.
The range is vast, but it is not all types of software.
.nh 3 "Projects for which Aegis' Testing is Useful"
.LP
For many years there have been full\[hy]screen applications on
text terminals.
In more recent times there is increasing use of graphical interfaces.
.LP
In developing these types of programs it is still possible to use Aegis,
but several options need to be explored.
.nh 4 "Testing Via Emulators"
.LP
There are screen emulators for both full\[hy]screen text and X11 available.
Using these emulators,
it is possible to test the user interface,
and test via the user interface.
As yet,
the author knows of no freely available emulators suitable for
testing via Aegis.
If you find one,
please let me know.
.nh 4 "Limited Testing"
.LP
You may choose to use Aegis simply for its ability to provide
controlled access to a large source.
You still get the history and change mechanisms,
the baseline model,
the enforced review.
You simply don't test all changes,
because figuring out what is on the screen,
and testing it against expectations, is too hard.
.LP
If the program has a command line interface,
in addition to the full\[hy]screen or GUI interface,
the functionality accessible from the command line may be tested using Aegis.
.LP
It is possible that "limited testing" actually means "no testing",
if you have no functionality accessible from the command line.
.nh 4 "Testing Mode"
.LP
Another alternative is to provide hooks into your program
allowing you to substitute a file for user input,
and to be able to trigger the dump of a "screen image".
The simulated user input can then be fed to the program,
and the screen dump (in some terminal\[hy]independent form)
can be compared against expectations.
.LP
This is easier for full\[hy]screen applications,
than for X11 applications.
You need to judge the cost\[hy]benefit trade\[hy]off.
Cost of development,
cost of storage space for X11 images,
cost of
.I not
testing.
.nh 4 "Manual Tests"
.LP
The Aegis program provides a manual test facility.
It was originally intended for programs which required
some physical action from a user,
such as "unplug Ethernet cable now" or "mount tape XG356B now".
It can also be used to have a user confirm that some
on\[hy]screen activity has happened.
.LP
The problem with manual tests is that they simply don't happen.
It is far more pleasant to say "run the automatic tests" and go for
a cup of coffee,
than to wait while the computer thinks of mindless things to ask you to do.
This is human nature:
if it can be automated, it is more likely to happen.
.nh 4 "Unit Tests"
.LP
Many folks think of testing as taking the final product and testing it.
It is also possible to build specialized unit tests, which exercise
specific portions of the code.  These tests can then be administrated
by Aegis, even if the full\[hy]blown GUI cannot be.
.nh 3 "Projects for which Aegis' Testing is Least Useful"
.LP
Another class of software is things like operating system kernels
and firmware;
things which are "stand alone".
This isolated nature makes it the most difficult to test:
to test it you want to provide physical input and watch the physical output.
By its very nature,
it is hard to put into a shell script,
and thus hard to write an Aegis test for.
.LP
The above chapter was written in 1991.  At this writing (1999) there are
projects like \(muLinux and operating systems like VxWorks.  These are
all embedded, and all have excellent network and download support.
It is entirely possible (with design support!) to write automatically
testable embedded systems.
.nh 4 "Operating Systems"
.LP
It is not impossible,
just that few of us have the resources to do it.
You need to have a test system and a testing system:
the test system has all of its input and outputs connected
to the outputs and inputs of the testing system.
That is,
the testing system controls and drives the test system,
and watches what happens.
.LP
For example,
in the olden days before everyone had PC and graphics terminals,
there were only serial interfaces available.
Many operating system vendors tested their products by
using computers connected to each serial line to simulate "user input".
The system can be rebooted this way,
and using dual\[hy]ported disks
allows different versions of a kernel to be tried,
or other test conditions created.
.LP
For software houses which write kernels,
or device drivers for kernels,
or some other kernel work,
this is bad news:
the Aegis program is probably not for you.
It is possible,
but there may be more cost\[hy]effective development strategies.
Of course, you could always use the rest of Aegis,
and ignore the testing part.
.LP
However, Aegis has been used quite successfully to develop Linux kernel
modules.  With suitable \fIsudo\fP(1) configuration to permit access to
\fIinsmod\fP(1) &co, developers can write test scripts which load device
drivers, try them out, and unload them again, all without universal
\fIroot\fP access.
.LP
Also, the advent of modern tools, such as VMware, which allow one
operating system to "host" another, may also permit straightforward
testing of kernels and operating systems.
.nh 4 "Firmware"
.LP
Firmware is a similar deal:
you need some way to download the code to be tested into the test system,
and write\[hy]protect it to simulate ROM,
and have the necessary hardware to drive the inputs and watch the outputs.
.LP
As you can see,
this is generally not available to run\[hy]of\[hy]the\[hy]mill software houses,
but then they rarely write firmware, either.
Those that do write firmware usually have the download capabilities,
and some kind of remote operation facility.
.LP
However, this omits the possibility of not only cross compiling your
code for the target system, but also compiling your code to run
natively on the build system.  The firmware (in the host incarnation)
then falls into one of the categories above, and may be readily tested.
This does not relieve you of also testing the firmware, but it increases
the probability that the firmware isn't completely useless before you
download it.
.LP
By using an object oriented language, such as C++, the polymorphism
necessary to cope with multiple environments can be elegantly hidden
behind a pure abstract base class.  Alternatively, by using a consistent
API, you can accomplish the necessary sleight\[hy]of\[hy]hand at link time.
.LP
The unit test method mentioned earlier is also very useful for firmware,
even if the device "as a whole" cannot be tested.
.\" vim: set ts=8 sw=4 et :