How to manage embedded
software development for startups — with Aegis How to manage embedded
software development for startups — with Aegis This page was inspired by the following article:
Hardison, O. (2004), How to manage software development for startups, Embedded Systems Programming, 15-Nov-2004.
Although software development processes are numerous and well documented, they don't always fit the time and resource constraints of an embedded systems startup company. Often a startup's software team follows a leaner path to achieve their goal: get a product out on time with no major defects. These teams frequently use open source tools and a mix of hybrid software development life cycle techniques, but there are as many nuanced processes as there are embedded systems startups.
What works? This article provides a set of software development processes that work well in the context of an embedded systems startup. These include:
While some of these steps may seem obvious, each step described is vital to achieving the stated goal.
Your software configuration management (SCM) tools, and bug- tracking database, should preferably have a web based query and reporting interface. It is not necessary to export this interface to the world, having it available on the company's intranet is sufficient.
You should also have project plan documents (source controlled in the same repository as the source code) that contain the overall structure of each major module within the product. The project plan should point out the changes to be added to a module in each release. This information reduces architectural and release-oriented questions that might otherwise be foisted on engineers during the cycle, thus freeing them up from low-priority interrupts. One of the major jobs of engineering managers is to protect their engineers from external information and feature requests. By having all pertinent information available online, a manager will reduce some of these requests; others can usually be handled by the manager himself.
For example, you may use the Linux model, and have odd numbered branches be development branches, and even numbered branches be release branches.
Alternatively, each change can be assigned arbitrary text attributes, and a “BugId” attribute is not only searchable, but will automatically appear in change set listings (use a lower case name if you don't want your change set lists polluted).
It is also simple to set up an email notification for every integration, not only to the software development team, but also to the documentation and quality assurance teams.
Once you have two or more developers, it is a good idea to turn on Aegis' code review features, which ensure that every change set has been code reviewed before it is integrated. Aegis supports many tools for code reviews, including tools such as tkdiff for color coded GUI code reviews.
Aegis also provides after-the-fact differences via the command line, or via the web interface (see the Aegis web site for an example of the information available about each change set).
Aegis attempts to ensure that all change sets integrated into the project “work”. This usually includes some concept of “compiles successfully and passes some tests”.
The project build is configured with an automated build command which is required to report success before your change can be put up for code review and eventual integration. This tends to dramatically reduce the number of commits which result in the entire development team being unable to build the product.
Because every integration builds the code, there is no need for the nightly automated build, because it has already been done. The results are available for all to see (and use, recalling that it works) in the branch's baseline.
The email for each change set should include a listing of all files integrated into the branch. In this way, engineers know what parts of the system have changed or what bugs were potentially fixed. To increase the report's usefulness to QA especially, the generating script should include the configuration identifier.
The build process typically produces a cross compiled image of the software to be downloaded into the target system. A very useful approach is to have the build process also compile a native executable for the build host.
Such an executable, while not cycle-identical to the target (in fact, it will frequently be considerably faster, and a different CPU) can be automatically tested via scripts on the build host, for very little additional engineering effort.
For example: by using conditional C macros for all I/O accesses, the target code accesses memory directly, but the build host version hands the access off to a device simulator, effectively allowing you to exercise the logic all the way to the bottom of your device drivers. When you run the executable, the console can be attached to the debug port simulator, giving an authentic (and scriptable) interface to the software being constructed.
Unit tests are equally useful on the build host, where they only contain an isolated portion of the complete code, and not sufficient operating system or run time environment to be viable on the target system.
Aegis provides facilities for supporting optional or mandatory tests accompanying each change set. By using automated tests in this fashion, many bugs are caught and squished long before the code is downloaded into the target system for real world testing (often, for small startup, tediously manual).
These tests accumulate in the project, and there is a simple Aegis command to run the accumulated set of tests in a developer's private work area before putting a change set up for review and eventual integration. Projects rapidly accumulate such tests, and it can become time consuming to run them all, so a subset can be designated as a “smoke test” and a shell script (source controlled, of course) made available to developers to run them.
For added insulation against a rapidly changing development branch (or one that project management has decided to “break” for a short time), a QA branch can be opened, with regular updates from the development branch, either via Aegis branch inheritance mechanism, or more manually via Aegis' various branch and change set cloning and reproduction tools.
Like any other branch, the usual build-test-review- integrate cycle is performed, giving opportunities for email notification, but also automatic on-demand builds of the product.
Release and support branches are most easily handled by ending the development branch (a branch is just a super change set in Aegis) and creating two sister branches, and even numbered branch for release and support, and an odd numbered branch for ongoing development and QA.
Change sets are promoted to the QA branch on a regular basis so that QA may perform their test suites and validate any fixed bugs. This can be automated using Aegis by having the QA branch be a child branch of the development branch. By building and integrating an empty change set, the QA branch's baseline inherits all of the change sets applied to the parent (development) branch since the last QA integration build. As a rule of thumb, a QA import is appropriate once a week so that QA has enough time to run their test suites and the queue of fixed bugs doesn't grow too large. The frequency of QA imports depends on the size of the project.
Ideally, QA will write automated test scripts so that bugs can be reproduced, and the test scripts are imported into the development branch to be run by developers to make sure that once a bug is fixed, it stays fixed. Maybe it is important enough to add to the smoke test script.
RC branches should be separate from development branches; ideally you can end the development branch, and the new RC branch will inherit all changes. Incomplete change sets that will not make it into this release can be cloned onto the new development branch just before the old development branch ends.
It is a good idea to archive RC build products (such as flash images and user documentation sets) so that a precise and exact copy of the user package is readily available. (Often necessitated by statistical FPGA compilers, which never produce the same answer even when given the same inputs). They're extremely useful for testing older releases and release candidates for the existence of some classes of difficult bugs.
Finally, both QA and RC builds should have their deltas named in Aegis so they can be reproduced by name, not just by the ID buried in the executable. Having delta names also helps generate the various reports so that engineering team and QA can see what fixes made it into each baseline.
The first part of the daily bug meeting should be devoted to the bug review. The meeting coordinator (either the project manager or the engineering manager in charge) runs a projector that displays a summary and individual bug reports. The latter part of the meeting should look at the bug queues (new, open, fixed) and outstanding issues to determine how the release is progressing along. The benefits of a regular review are many:
Note that some projects are so small that this meeting can quite profitably be replaced with the project manager looking at new bugs since yesterday and assigning them priorities and dispatching them to the responsible developer by email. Even in large organizations this can be very effective. The point is to deal with bugs rapidly, rather than let them accumulate to just before a release when you have no resources to spare to fix them. If you have to have a daily meeting, consider doing it in a room without chairs, it is amazing how short meetings can be when held this way.
In small startups, this meeting can be very small and frequently has only the team leader or project manager representing the developers (and he's up-to-date because he reads the integration email notifications) and QA and marketing.
Aegis can provide histograms of change sets making their way through the process, or you can mine Aegis' meta-data and make your own. Note, however, that this is implementation focused.
Another source of data, and probably more issue focused, is the bug state information from your bug tracking tools, such as Bugzilla. Statistics like these can also be graphed automatically on a web page through some CGI scripting.
Along the way you might consider some additional questions. Have you started on a maintenance release? Have you checked with your sales or support engineer (quite a valuable and underutilized resource for internal status checks) on how the release is being accepted? Has anyone considered a customer survey to see if the release produced as it promised or if it missed the mark?
The software development process described here (automated builds, full code reviews, host based automated testing) works, even when (especially when?) there is only a small number of developers, and it scales to accommodate your growth. Aegis does most of the work, releasing developers from paper-work hell. The automation of builds and the validations provided by tests and code reviews give you needed stability so that your team can focus on actually making your product.
 | The beautiful graphics on this web site are by Grégory Delattre. | Return to the Aegis home page. |
|