At this point, a crafty programmer can use subtle techniques which anticipate the sources of error and enable them to be identified and corrected in a predictable and orderly manner. In a famous discussion, Donald E. Knuth wrote of how he was able to greatly reduce the time it took to debug a new compiler by anticipating worst-case test conditions and "stress testing" by use of the adversary method. We employ similar techniques by testing the mechanisms used in the kernel separately from the vast body of code that is the kernel. This also has the added benefit of differentiating problems in the code from compiler and assembler bugs that are almost certain to be present. This is not a method that guarantees success, but it is much better to seek out trouble rather than wait for it to find you.
Serious thought was given to implementing a tiny debugger to facilitate this stage of the port. PC debuggers were also examined as a possible tool to ease this effort. However, it was determined that the effort to keep the tools concurrent with the rest of the project would be too costly for too little advantage. This proved, in retrospect, to be the correct strategy for 386BSD, since most of the bugs we encountered were either inherent to our naive assumptions regarding the 386 instruction set or to silent "features" of the particular version of the GNU GAS assembler used, and as such would have affected the debugger as well as the operating system kernel we were porting. However, an appropriate debugging tool might have cut our development time and would have been especially useful if we were contemplating many ports to other architectures.
In practice, an appropriate tool for kernel debugging should afford little impact on the absolute environment. It should allow for source-level debugging (now considered a bare minimum requirement) and should leverage existing development platforms as much as possible. Van Jacobsen's "kernel GDB," for example, is derived from GNU's GDB debugger. It uses a small stub routine in the kernel and a serial line to communicate with a cross-host running the debugger. Other debuggers probably exist that exhibit these qualities, but we are unaware of them.
With this article, our port is moved from the conceptual to the tangible world. This discussion, while by no means complete, addresses many of the mechanisms necessary for 386BSD kernel functionality. It is tantalizing to watch as the basic mechanisms come together, and one tends to think of what remains as mere bookkeeping. If this were true, operating systems programming techniques would more closely resemble those used by applications programmers. They do not, however. While it may appear that the end of the project is nearing, what is actually occurring is merely the first battle of a long and involved war. Again, to use our mountain climbing analogy, we clear the foothills only to be faced with the mountain range itself. In other words, we are continually challenged with a new class of obstacles.